.
Page 1 of 2 12 LastLast
Results 1 to 15 of 19

Thread: Observations on the Beta Testing Process

  1. #1
    Join Date
    Nov 2006
    Location
    United States of America, TN, Memphis
    Posts
    1,829
     

    Default Observations on the Beta Testing Process

    I have been reading all the messages regarding issues and problems with build 114800. I am running Trainz TR2019 plus. I guess I am fortunate in that all the routes I run work without a hitch. I was a beta tester so several issues along the way, so I submitted a bug report and they got fixed.

    However I probably use only 30% of the functionality of Trainz so some of the issues identified would not even cross my radar when testing. It may be the same for other beta testers. As an example there were several apparent problems with in-cab controls. I have never used them so I didn't think about testing these.

    There apparently were problems with reversed engines, but that apparently only affected people not using Trainz plus.

    I don't know how many people are involved in beta testing. I do know that several weeks before build 114800 went public there were no major issues discovered and reported (show stoppers), hence the decision to go public. So one conclusion is that build 114800 is performing well for the beta testers.

    Another conclusion may be that the beta testing process needs to be broadened to get a more meaningful sample of users who have different versions of Trainz (Plus, Platinum, Basic). Also there may be a need to include people who are really pushing the game to the limits and would be the most likely to find errors in scripting etc. One of the big issues identified and fixed was the portal issue which exist in build 111951. Along with this issue was a problem with the Central Portal Control, which was identified and fixed.

    Trainz is a very complex simulation program and N3V has done a great job in moving it along. I doubt whether the programmers spend a lot of time building and running routes. They need more help in identifying these issues so that they can be fixed. The question really is how does that happen?

  2. #2
    Join Date
    Nov 2006
    Location
    United States of America, Alabama, Birmingham
    Posts
    3,719
     

    Default

    I agree with most everything you say. But I have to point out that release candidate versions are made public in the forum for anyone to test. I suspect that few people bother to do so because they don't want to mess up their working install. That is certainly the way that I feel and I don't bother to patch until others have reported their experiences. I believe that proper testing is hard work and I thank all the users that are willing to pay for the privilege of being able to test. I'd rather spend my time enjoying the game. Now I understand that N3V can't fix what they can't reproduce and I imagine they get a lot of bug reports that fail to provide enough detail for them to reproduce the problems. That is always going to be a problem in an open beta system.

    William

  3. #3
    Join Date
    Nov 2006
    Location
    Australia, NSW, Sydney
    Posts
    8,312
    Blog Entries
    3
     

    Default

    I was also a Trainz Plus SP4 beta tester and before I upgraded to Plus I was a TRS19 Platinum Edition SP beta tester.

    You have pointed out several drawbacks of the beta testing system. Obviously the more beta testers you have the more likely you are to discover problems. If potential testers see it as "risky" i.e. it will interfere with my route development, or time consuming then fewer users will apply.

    One issue I found is the advice that beta testers often get is not to undertake any development work during beta testing. Yet that is an area which has the most potential to discover problems - when you are "pushing the envelope", so to speak. When I was beta testing TANE SPs I ignored that advice and continued with development work in the beta version. I did find bugs that no-one else seemed to have reported but I also lost a lot of work and time and had to go to my backups - some of which were corrupted by one of the early SP betas. A lot of swearing (at my own stupidity) and frustration, but I knew the risks when I became a beta tester. I did not blame N3V for that.

    With TRS19/Trainz Plus I decided to play it safe. I did all my development work in the stable (non-beta) TRS19/Trainz Plus on one computer and copied the work (via .cdps) to the beta on another computer and tested what I had done there. I discovered a few minor bugs. Only when the final SP4 release came out which made me continue my development work in SP4 did I discover more serious (or annoying) problems.

    Another issue is reproducibility by others. There have been a few bugs I have reported in TANE and TRS19 where it seems that my reports were the only ones. Most were acknowledged as bugs by N3V QA but it is highly likely that because no-one else had experienced them they were given a lower priority - and I can fully understand this. You prioritize your scarce resources to fixing those issues that are seen as more common or urgent. Plus they were bugs encountered in very obscure and probably seldom (if ever) used situations. I am pleased to report that at least one of them has since been fixed. It takes patience!

    One issue I discovered early in TRS19 beta testing I did not report as a "bug", it seemed to be more of a "curiosity" than a "bug". I have since reported it and provided more documentation to N3V QA but I did document it on the Trainz Wiki. Only recently (today) did I get feedback from other users that prompted me to report it. Perhaps if I had acted sooner it may have been fixed in SP4, or not!

    I tend to be timid, even when beta testing, and I don't always "push the envelope" as much as it should be pushed. But now that I have two computers with enough oomph to run Trainz Plus at somewhere near its potential, more or less, that may change in future.

    My thoughts and experiences.
    Peter
    Trainz Plus (TRS19) build 114800 (SP4)
    Trainz Plus (TRS22) build 115628 (beta)
    Trainz Simulator 3 (Mobile)

  4. #4
    Join Date
    Nov 2006
    Location
    United Kingdom, Cheshire
    Posts
    2,508
     

    Default

    I don't have Plus, just a basic install.
    On one of my 3 working installs (each on its own SSD) I updated to SP4 because it had been Beta tested by those that could and I put my trust in N3V that it was good enough for release to the public.
    I now find myself with a faulty SP4 and reporting those faults that I have found, so I am not a Beta tester I am now a 'conscripted' Public Release tester.
    Don't get me wrong, I am happy to report what bugs that I do find, but, maybe if those members with a basic install (there are many heavily involved users of TRS19 that are) were allowed to beta test, we could have helped N3V before SP4 was released.
    I tried plus and did not find the UDS useful (in my opinion) and I very rarely purchase DLC.
    I do buy a 3 Month FDC to help the upkeep.

    Beta testers know the risk factor and need to take measures to minimise the risk, but now it's risky for the public to install a Service Pack and that's not good for N3V's reputation.

    cheers,
    Graham


    Member of TCWW & Trainz.de

  5. #5

    Default

    Beta testers have no access to the source code. Many have little or no idea as to graphic or physics engines, databases or compilers. They cannot be sure which routines affect other (sub)routines. They are expected to operate as a consumer, not a coder and therein lies the issue. When bugs are discovered, unless they are in specific coder lingo, and meticulously described debugger outputs, it falls on deaf ears. I was quick to report the reversed assets bug and got flipped off as having a bad system and emotional issues by fanbois that love to hear themselves talk.

    I won't make that mistake again.
    Algoma Central - Canadian Pacific - Canadian National - Esquimalt and Nanaimo - BC Rail

  6. #6

    Default

    Moreover, how this slipped past non-Plus users is negligence. It's disquieting to know dev didn't see something so blatantly obvious the instant Surveyor is loaded.

    Coder v bean counter. Who's to blame? My guess, it's almost always the latter.
    Algoma Central - Canadian Pacific - Canadian National - Esquimalt and Nanaimo - BC Rail

  7. #7
    Join Date
    Nov 2006
    Location
    Netherlands
    Posts
    896
     

    Default

    Many things should have been caught even before it is in a beta
    -Test-track, this takes 2 minutes to find/test for anyone
    -Sideways moving sliders in Driver list that overlaps text
    -the shock movement in camera mode 3
    -the F2 bugs, it requires 1 pc at n3v with standard/regional (or a script/account to limit rights)
    -New water although nice is still under development, needs serious warnings so users don't ruin years of work


    I never blame beta testers, but thank them for all hard work
    It would help if there was a standard quick check list, that every beta tester could do in lets say 2 hours
    and besides that detailed info what has changed, so a specific build beta test can concentrate there


    I prefer to be at the begin of the line (generate/propose ideas) and at the end (Omega tester)
    greetings GM

  8. #8
    Join Date
    Nov 2006
    Location
    United States of America, Massachusetts, Haverhill
    Posts
    30,444
     

    Default

    I tested as well, although, not as the furiously as I normally do due to outside obligations this time around. For my testing, I have a duplicate of my working setup that I update to the beta version du jour. This gives me a chance to vet out any faulty content that may appear on the horizon later, known issues not counted of course. Having a duplicate setup means I can test to my heart's content without worrying about messing up my working installation. This setup also allows me to compare operations so I can see if an issue occurred in a previous version, or is a new one in the current test.

    As Paul pointed out in the original post, there are many things reported that we never saw, and likewise we never can catch everything. This is why we need as many eyes on the beta and problems reported to get them on the repair list.

    I agree that there should be more internal testing prior to release to beta. If the code that affects a subsystem has been changed, test it before moving on. This is a fault of N3V's internal process and that needs to be addressed. As I said before, this is pretty obvious that the coder(s) do not use the product in any fashion. Back in the Brew Crew heyday, the developers were required to build routes and put the product through its paces. This didn't have to be an elaborate setup, but only a way to test the functions such as splines, engine interiors, the Test-track, and other early pre-beta anomalies. I know N3V is short-staffed, but that's no excuse for a poor product release. That's like bringing a car to a repair shop and finding the oil-cap missing. When asking the mechanic about that, he says he's busy and doesn't have enough employees.
    John
    Trainz User Since: 12-2003
    Trainz User ID: 124863
    TRS2019/Trainz-PLUS: 114800
    TRS22 Beta tester

  9. #9
    Join Date
    Nov 2008
    Location
    Crewe, Cheshire, UK
    Posts
    17,296
     

    Default

    Quote Originally Posted by dleblanc View Post
    Moreover, how this slipped past non-Plus users is negligence. It's disquieting to know dev didn't see something so blatantly obvious the instant Surveyor is loaded.

    Coder v bean counter. Who's to blame? My guess, it's almost always the latter.
    IMO we need more Non Plus users to actually test rather than suddenly appear on the forum complaining about things not being picked up by beta testers, those who think they could do a better job need to step up and participate in testing so these things get dealt with quicker. It's easy enough to set up a second install for testing so you don't break anything in existing routes etc, if you have the disk space of course.

    Need to remember that N3V does not have a large army of coders such as the major software houses have at their disposal, heck even Microsoft can't release anything without bugs, first Win11 cumulative update had 74 bug fixes! including major issues with Ryzen CPUs causing a performance hit of up to 15%, thankfully now fixed however wasn't on the released build.

    I adopt the principle that just because something doesn't affect me, doesn't mean there isn't a problem for others, we all have a vast multitude of different assets, routes, various scripts, preferred methods of doing things and even hardware so one should not dismiss any reports of problems out of hand, just because they are not affected.
    Malc


  10. #10
    Join Date
    Nov 2006
    Location
    Netherlands
    Posts
    896
     

    Default

    IMHO, all versions of Trainz (regional, standard, plus, platinum) are basicly the same code
    the difference is scripted with Product rights.
    You start the game, you get checked what product rights you have according to your account
    and the game selects a subset of features enabled/diabled.


    If there would be an option to select a lower product right
    a plus user could emulate how a regional/standard would be (no UDS but F2 Quickdrive)
    limiting beta to plus only, in the end gives us the current situation


    Many bugs/problems were detected during beta FI:
    -swithes in cabins, Rbrooks
    -wheel rotation p_dehnert


    if not taken serious at that time, it leads to what we have now
    Something does not go, by winking it away, saying this is current design, or say content issue or local issue
    it lingers on and in the end make users leave or frustrated.

  11. #11
    Join Date
    Nov 2006
    Location
    United States of America, Massachusetts, Haverhill
    Posts
    30,444
     

    Default

    It was mentioned that it's important to log the issues in the bug-report, and if that's not done then the issue won't be caught. This might have been part of the problem with catching these errors, or bugs in general, because people only post them here and not the official bug report. The problem here is the developers don't read the forums because they're busy chasing bugs.
    John
    Trainz User Since: 12-2003
    Trainz User ID: 124863
    TRS2019/Trainz-PLUS: 114800
    TRS22 Beta tester

  12. #12
    Join Date
    Nov 2006
    Location
    Netherlands
    Posts
    896
     

    Default

    I tried to give an example of developing something in the Thread Set-Driver-Condition-Rule
    Not only share everything that has changed, but also give insight in the thoughts behind it
    then read and take all feedback in, not only from the thread on this forum, but also on other forum
    and mail and PM's,


    ofc. for a much more complex development as Trainz this is harder.
    A bug report in this case is good, but other users can't see IF it is reported, HOW it is reported
    and what has been done with it or the reply it has gotten. hope this makes sense.

  13. #13
    Join Date
    Nov 2008
    Location
    Crewe, Cheshire, UK
    Posts
    17,296
     

    Default

    If you report a bug already reported that's being dealt with you get a reply to that effect and it's reference number, leastwise I have. However IMO the more that report the same bug the more likely it is taken as serious instead of a one off or operator error etc.
    Malc


  14. #14
    Join Date
    Oct 2018
    Location
    USA - North Carolina
    Posts
    2,193
     

    Default

    It would be interesting to know how many Plus beta users actually beta test. I suspect there is a large number who go beta just to get the "latest features" and who do no testing at all. I'm not the most "aggressive" tester there is and I have turned in reports. But it seems that it's always a select FEW people who report and/or mention "bugs" in reply to Tony here in the beta forums. It seems to me that Plus (beta) is leveraged more to help with revenue than to help find more beta testers. But I could be very wrong. I have no idea how many different names appear on Bug Reports vs how many users of the Plus Beta stream there are.
    Gold/Plus
    22+ beta 115628;
    19+ SP5 beta 115653; 19-Std SP3; Tane SP4
    Win10, GTX1060-6, 1TB SSD(C), 2TB SSD(D)


  15. #15
    Join Date
    Nov 2006
    Location
    Swansea, Wales.
    Posts
    32
     

    Default

    Quote Originally Posted by wreeder View Post
    I agree with most everything you say. But I have to point out that release candidate versions are made public in the forum for anyone to test. I suspect that few people bother to do so because they don't want to mess up their working install.... That is always going to be a problem in an open beta system.

    William
    Hi William - I didn't know that release candidate versions were made public in the forum for anyone to test. Where does one find it or was it only available to plus users/subscribers?
    I would say that the current beta testing arrangement is a 'closed' beta system, available to subscribers only.


    Quote Originally Posted by euromodeller View Post
    I don't have Plus, just a basic install.
    On one of my 3 working installs (each on its own SSD) I updated to SP4 because it had been Beta tested by those that could and I put my trust in N3V that it was good enough for release to the public.
    I now find myself with a faulty SP4 and reporting those faults that I have found, so I am not a Beta tester I am now a 'conscripted' Public Release tester.
    Don't get me wrong, I am happy to report what bugs that I do find, but, maybe if those members with a basic install (there are many heavily involved users of TRS19 that are) were allowed to beta test, we could have helped N3V before SP4 was released.
    I tried plus and did not find the UDS useful (in my opinion) and I very rarely purchase DLC.
    I do buy a 3 Month FDC to help the upkeep.

    Beta testers know the risk factor and need to take measures to minimise the risk, but now it's risky for the public to install a Service Pack and that's not good for N3V's reputation.

    cheers,
    Graham
    Hi Graham - I am also just a basic install user. The subscription model is not for me. Tried it once and found no benefit. I don't fully understand N3Vs reasoning behind their beta testing arrangement. It's a bit of a cheek to expect customers to pay a monthly fee to be able to test their product. I am in limbo at the moment in that my copy of Trainz 19 was purchased via Steam. Steam does not give you the option to reject an update and does not permit you to backup the program until the update is complete. So at the moment I am stuck and have not played the game since SP4 was released for fear of becoming a 'conscripted public release tester' like yourself. I have manually backed up my Trainz install and Data folders just in case.

    Quote Originally Posted by clam1952 View Post
    IMO we need more Non Plus users to actually test rather than suddenly appear on the forum complaining about things not being picked up by beta testers, those who think they could do a better job need to step up and participate in testing so these things get dealt with quicker. It's easy enough to set up a second install for testing so you don't break anything in existing routes etc, if you have the disk space of course.
    Hi Malc - Can non-Plus users beta test then? If so, how do we go about getting added to the beta test team?


    To dleblanc - you were quick to report the reversed assets bug. If this forum engine supported thumbs down it would have got one from me for the person's reply who flipped you off.


    We know now that Tony Hilliam has provided a link for the SP3 installer. This indicates to me the serious defects in the SP4 release. I cannot remember having a link to a previous installer before. Why not make the installer available from the MyTrainz dashboard instead of tucking it away in a forum? In fact all versions, base and service packs should be available to the purchaser. It should be for the end user to decide which version works best for them.

    To Tony - thank you for the SP3 link, but will this work for Steam users? If not then what do we do? I'm stuck in limbo at the moment.

    Best regards to everyone,

    John

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •