Asset Database Backup: Wow! Over 20,000 files!

[ianwoodmore]

@MasterTracklayer

Hi,

I may be able to help you if your previous trainz version is not too old.
I'm at Trainz Plus build 123794.

In your old Userdata folder, at the bottom, you may see a file 'keywordsbackup.dat'
If so you are in luck.

1.First, save your latest Trainz build version of this file.
Substitute the same named file from the old Trainz build.

When you have transferred your assets from the old Trainz

In the new Trainz, select Trainz Settings then Dev.
Click in box next to 'Enable advanced debug tools'.

Re-boot new Trainz.
In Content Manager on top menu under Developer you should see two drop down items:

a. 'Run TrainzUtil Command'.
b. 'Rebuild Database'. Usually known as DBR.

Hold down Ctrl and Rebuild Database. Usually known as EDBR.

 

[ianwoodmore]

@MasterTracklayer

Hi,

I may be able to help you if your previous trainz version is not too old.
I'm at Trainz Plus build 123794.

In your old Userdata folder, at the bottom, you may see a file 'keywordsbackup.dat'
If so you are in luck.

First, save your latest Trainz build version of this file.
Substitute the same named file from the old Trainz build.

When you have transferred your assets from the old Trainz.

In the new Trainz, select Trainz Settings then Dev.
Click in box next to 'Enable advanced debug tools'.

Re-boot new Trainz.
In Content Manager on top menu under Developer you should see two drop down items:

a. 'Run TrainzUtil Command'.
b. 'Rebuild Database'. Usually known as DBR.

Hold down Ctrl and Rebuild Database. Usually known as EDBR.

If you have many files, do this overnight.
New Trainz has to re-format in tzarc compressed files to reduce size of your database.

I'll get back to you shortly as I have an errand to run.

 

[MasterTracklayer]

ianwoodmoor, both installations are/were the latest available. Both are/were Trainz22. One was the latest Win7 build available a few weeks ago, and the other was the latest Win10 build yesterday. Neither database has that file, as far as I can tell.

Edit: I`m currently engaged in manually exporting over 5000 kuids (just the kuid itself, **NOT** the whole asset) into a .txt file for possible use in redownloading into the new database. I`m currently about halfway through the operation. So far, it looks like davesnow is the creator of an absolutely huge portion of those 5000+ assets.

 

[blueodessey]

blueodessy: Have you seen the edits to my original post? The algorithm mentioned in the first edit is designed especially to cope with this exact issue: per-file overhead is nearly constant while data transfer is usually a linear function of the size of the file. Conventional algorithms make no distinction and thus overreact to changes in file size. Mine considers them separately and issues estimates accordingly. Unfortunately, nobody uses my algorithm. I`ve been playing with computers for about that long too.

As for Airplane Mode, I`ve got that on my phone, but have never seen it in Windows. Regarding your other suggestions, I agree entirely, in principle, but I haven`t a clue how to do that without killing Windoze 10. As it happens, it only took approximately two hours to copy it all, which is nothing compared to, oh, about twelve to fourteen or so hours to download Trainz22. Twice: Remember that I was uncertain which Windows I am running until I saw a post that mentioned winver, so I downloaded the Win7-compatable version, just to be safe, then downloaded the current one once I knew my OS was compatible.

Butler57: Thank you; that was (obviously -- see above) my second thought regarding the meaning of the reference.

Everybody: How do you manage everything? Like I said earlier, I made heavy use of editing custom keywords directly in Content Manager without editing the assets themselves. No longer possible. Simulating it with Pick Lists, while possible, looks likely to be unwieldly. I would filter on my custom keywords in such a way that miscategorized assets would appear in the correct category filters and not in thee wrong ones. I used this to make a finer-grained filter than Content Manager was able to handle without it. Unfortunately, this took LOTS of custom filters, and while running them was fast enough, editing them has always been, and still is, glacially slow. -- and the number of "items" that Windoze counted included all the files and all the directories, all counted up into a single number. Windoze has no idea of an item as Trainz counts them.

Edit: with the updated Trainz22 downloaded and my meagre asset collection backed up, I`m about to try installing the new version. Wish me lock, people!

Mastertracklayer ....My bad, I did see your Algorithm, and realized what your comparing. After further with Windows showing a full Day estimate to do the copy command, dummy me, didn't recall, that Windows is notorious for inaccurate timing estimates.

As per:

Everybody: How do you manage everything?

I store special searches by Name, and I store a lot of Asset Makers in my CM, helps me find them quickly. In picture below, you'll see Author Names with - in front of name, this allows for it to show at the beginning of List. + in front moves items lower. I do need to do some more House Keeping to sort out the rest of the listings.



I use a slew of Keywords, for finding different types of Assets by specific Grouping to the key word.

Words below are just a few of the ones helpful to me in my Query for particular assets.
Junk, Trona, Compressor, Salvage, 2021, Zona, Tram, Slug, Colton, Rebar, Sharknose,New Zealand, Welsh, Quarry, Derelict

Iceland, Coffin, Tugboats, Steamboat, Coal Mine Basic, Power Station, Lumber Mill, Forestry Basic, Oil Rig, Oil Refinery, Stone,

 

[MasterTracklayer]

blueodessey: I didn`t realize that your name would paste as **formatted** test, but it did. I don`t know if this text will retain its formatting when I submit it, but I`ll find out. If it does: Apologies everyone, but I didn`t know.​

Regarding the algorithm: No big deal; I occasionally have "brain farts" myself, more frequently as I age.

Regarding content organization methods: I used to use my **own** keywords, which I assigned in such a manner that a search for a particular asset type would start with a generic search for the type, filter OUT any assets tagged as being **not** the type they clamed, and add in any that were tagged as being that type despite claiming that they were. I didn`t much care **who** had written the asset, and mine were **all** from the DLS, so no need to filter by where I got `em.

For example, start with all assets claiming to be locomotives, remove all that had proven to not be locomotives after all, and add in all that have proven to be locomotives despite claiming to be something else.

I also expanded this scheme to classify scenery as what type of scenery it was at levels deeper than the built-in classification system could dream of. For example, it would have been easy to generate a filter to list all splines of crops instead of just all splines of flora, and it would include crop splines that thought they were track, and not include any track that thought it was a crop spline.

Alas, I cannot do that any longer, and faking it with pick lists would be "difficult" to maintain, mostly because pick lists are so bleep good for other uses.

Status update: I have finished hand-generating a reasonable facsimile of the DLS content that I expect to redownload, and saved it as a *.txt file. I will open my own assets for editing, and copy them somewhere else. I will then revert my Win10 Trainz22 to the actual Win10 Trainz22 content database and start TRYING to download the still-missing Win10nversions of the "packaged" and "built-in" assets. While that is taking place, I will leave these forums for a while, though I expect to return before it is finished. Despite having an active First Class Ticket at the moment, my connection is flakey enough that I may be unable to download ANY of it via this internet access point. If so, I will attempt to download it via another connection point that I have occasional access to. DLS assets can wait until that is finished.

So: goodbye for a while, but I will be back.

 

[ianwoodmore]

ianwoodmoor, both installations are/were the latest available. Both are/were Trainz22. One was the latest Win7 build available a few weeks ago, and the other was the latest Win10 build yesterday. Neither database has that file, as far as I can tell.

Edit: I`m currently engaged in manually exporting over 5000 kuids (just the kuid itself, **NOT** the whole asset) into a .txt file for possible use in redownloading into the new database. I`m currently about halfway through the operation. So far, it looks like davesnow is the creator of an absolutely huge portion of those 5000+ assets.

Okay I'm back. WIN 7 or WIN 10 is not an issue in Xferring the assets, but may affect the DBR processing time if Windows decides to index them. I've not tried Xferring using TXT file. If you have few assets in your new instal it should be possible to replace the new Local and Original folders with the old ones. As I have a maxi database comprising ALL the DLS and many DLC I chose to Xfer the Local and Original folders along with assets.tdx and backupkeywords.dat. Those in the 'original' folder are unmodified, whereas the Local folder contains all the modified assets and some others. When you revert an asset in CM it uses the 'Original' copy. Even after an update in format this can be done. NOT recommended if you are impatient as will take several days and several DBR to clean up. Mind you, your number of assets being Xferred is much smaller than mine as I have nearly 700K assets (several Terabytes) listed in my TAD.


The keywords are stored in each asset in your old instal as listed in tag 'Category-keyword'. Thus, if you Xfer them as assets NOT as a TXT file they will survive, and can then be consolidated in assets.tdx using DBR function. But not if you DL assets to a new machine. The Backup function won't be apparent in CM until you do a DBR and trainzutil 'backup keywords'.

Yeah, Dave is a prolific creator and a good one at that. Excluding the assets you can't export and the obsoletes I have 6925 installed and only the 914 obsoletes on the DLS.

Suggest you build a custom CM filter in your newest Trainz version.

Save as _AUTHOR - davesnow. The leading underscore minimises false positives as CM uses partial word sorting. All creator's names lower case excluding Auran. This will group them in Alphabetical order. I use CAPS on group names to differentiate my custom filters from the default filters available in CM.

Suggested start as below, but I have hundreds of custom filters for analysing repairs. Tell me what you need, and I'll list them here.

Example

Author davesnow
On Download Station false or true
Installed false or true
Builtin false
Payware false
Packaged false
Obsolete false
Faulty false
Faulty Dependencies false
Missing Dependencies false
AND
Name (Name is all categories plus MISC. Alternatively use Category and whatever you're focussing on)

This is a maxi filter that you can switch false or true in various combinations for analysis. Some combo may not work. You may wish to simplify it to suit your needs.

You don't want to Xfer any 'Obsolete', but if you do, use delete after highlighting them. In my TAD I have 914 of Dave's.
You can't Xfer 'Builtin' - not authorised.
No point in Xfer 'Packaged' - Most of Dave's LEGACY assets are packaged. Some can be modified, and some can't, but as these are primarily aimed at Multiplayer Surveyor they will get reverted next time you do a DBR. Trouble is that some packaged assets have copies on DLS and some don't.

'Incompatible' means they are assets that are later TBs than your build. ie. Some beta or next build before it is released to the Community.

Incidentally, in the Game's new Surveyor 2.0 there is an on-line CM called Assets Palette. Your picklists will appear here too. I've not had much time to evaluate this as yet.

As I wasn't sure how familiar you are with CM, I decided to kill two birds with one stone, so to speak, as many others may wish to try this procedure. This detailed description of the procedure and various pitfalls is offered to the Community. Comments and Criticisms will be gratefully received.

See John Citron's similar procedure here:

https://forums.auran.com/threads/ar...from-trainz-2019-to-trainz-2022.176354/unread

 

[MasterTracklayer]

Status update: at the time of writing (NOT posting), I`m downloading West from Denver into the new installation`s database, 2418MB of 7094MB, 845 minutes remaining, and the download hasn`t aborted. Not yet, anyway.

ianwoodmore: I am quite comfortable with the Content Manager, having made extensive use of it with my "Trainz Simulator 2 pack", containing both the 2010 and 2009 releases in a single offering. As I mentioned earlier, those 5600+ kuids are just the kuid, and I`m not transferring anything with them; they are just a list of the kuids of DLS items that I will want to redownload, and I may choose to not redownload some of the older fixed-running-number rolling stock (from the days before ARN, when each different number had to be a separate download; I`ve collected some OLD stuff, for reasons of my own). The only stuff I want to **transfer** are my four (four!) assets, most of which are not really worth the bother, but which I have saved open-for-edit copies of anyway, to import into the new database, and my collection of filters, which will not be difficult to recreate, just tedious, thanks to how slowly Trainz manages the filters themselves. (They *run* just fine; they just edit/rename/save like a snail.) The items in the list I prepared do not contain any assets that listed as internal, broken, etc. Actually, I believe that I forgot to INclude the installed-from-DLS assets that had updates available; I may have to go back and add those to the list. All I was doing with the various status counts was commenting on what the Win10 Trainz22 thought of the assets that I had accumulated with the Win7 Trainz22. Not surprising that there would be a difference of opinion between the two versions of Trainz22, just reporting what that difference of opinion worked out as on mu machine; the last time I had looked under the Win7 version, there were NO obsolete, incompatible, or out-of-date assets listed, to the best of my memory, yet the Win10 edition said there were. Nothing to worry about; just expected, given that the Win10 version as a different version of some of the built-in assets and DLS access to others that would have broken under the Win7 executable. Naturally I had my filters set to exclude those to-new-to-use assets when performing the original DLS downloads with the older program. Other than asking how big people`s assets directories are, how they manage them, and how to migrate filters (all of which are still open questions), I`ve mostly been chronicling my oddesy (argh, can`t find a spelling for that word that my browser`s spell checker likes) for the benefit of others in similar situations. That said, while your comments haven`t helped me personally much, they are certainly a welcome addition for the edification of anyone else wo might read this thread.

Something puzzles me, though: Why would an asset that presumably did NOT list as incompatible in the older program list as such in the new one? Too OLD for the newer program, perhaps, instead of too new for the older?

P.S.: now 2909MB of 7904MB with 792 minutes remaining, and still no barfburgers!

 

[MasterTracklayer]

Update: still downloading, 4703MB/7904MB and counting, 603 minutes remaining.

Reason for new post: I`ve found and read the Wiki page for trainzutil.exe, and have learned that trainzutil can handle all of the keyword-manipulation now missing from Content Manager, EXCEPT managing the keywords from a GUI. My management scheme is still usable, just not convenient. Well, I`ll just have to find a way to work around that. trainzutil is a powerful beast, and I can wrap it in a Python-managed Tcl/Tk GUI of I need to. It will take some time, but it should be eminently doable. Theoretically, I could even recreate Content Manager itself this way. I may talk more about this later.

Update: still downloading, 4828MB/7904MB and counting, 590 minutes remaining.

I`m going to need to look into the possibility of downloading Trainz Content window items this way.

 

[MasterTracklayer]

<grumble><grumble><gripe><gripe><gripe>Arrrg! Other salty expressions.

Bleep, but the censored download of West from Denver has restarted -- FROM THE BEGINNING, and apparently all by itself, and is back up to over 1000 minutes remaining. I need a way to download this that can RESUME a download instead of RESTARTING it. Can anyone suggest anything? I fear my First Class Ticket will expire before I can visit my faster alternate connection point. Bletch. This leaves a taste in my metaphorical mouth similar to the one in my actual mouth, only worse. Gah!

 

[PhilChorusch01]

Advice! Backup Original, Local and Packages in a rar file. From C:\Users\Win10 Admin\AppData\Local\N3V Games\trs22 then inside the build xxxxxx folder! Save the rar file to Desktop.

Completely, uninstall TRS22. Then delete local saved data from C:\Users\%Username%\AppData\Local\N3V Games
Tip: Use Shift+Delete then OK, restart your computer.

Reinstall TRS22, then restore your Original, Local and Packages by extracting the rar file to C:\Users\Win10 Admin\AppData\Local\N3V Games\trs22 then inside the build xxxxxx folder

Launch Trainz | Click on the Developer menu | Rebuild database | Exit Trainz
Then relaunch around 30 seconds after you have exited Trainz..

Go into Content Manager 'Manage Content'
Filter:

Installed: True
Out of date: True

Right click on 'one at a time' Packaged, Packaged/Payware, Built-in, Modified, Obsolete. Then choose, List Assets Versions then you will be able to download the updated assets.
Any assets you have listed as Installed from DLS, Newer version available.. Select all of these in the list right click then choose download.

 

[MasterTracklayer]

That sounds like good advice, PhilChorusch01, perhaps even slightly overkill, but in my case, it is totally impossible to implement it, and it might not have helped me even if I could have. I tend to have weird setups where prepackaged help often doesn`t work. Besides, it is long since too late to do most of that: Win10 Trainz22 has already been installed over Win7 Trainz22 (this is on a Win10 machine; I downloaded and installed the Win7 Trainz22 before I knew which Windows I have; the entire Win7 Trainz22 asset database has been copied to a different location; Win10 Trainz22 likes the Win7 Trainz assets just fine, but expectedly disagrees with Win7 Trainz22 about their disposition; I`m currently downloading what should be the Win10 Trainz22 packaged assets (with network issues) into the Win10 Trainz assetbase with Win10 Trainz22; and anything that has to be done before any of that is flat out impossible.

Having been an occasional Trainzer for a couple decades or so, I`m competent to stumble through this and get most of it done without major issues, but some of the nuances, especially of things that have changed since my previous Trainz experience, can still make things a bit more difficult than they HAVE to be. <shrug>

Like I said to ianwoodmore: "That said, while your comments haven`t helped me personally much, they are certainly a welcome addition for the edification of anyone else wo might read this thread." I still hope that other people here can learn from my mistakes.

 

[blueodessey]

Update: still downloading, 4703MB/7904MB and counting, 603 minutes remaining.

Reason for new post: I`ve found and read the Wiki page for trainzutil.exe, and have learned that trainzutil can handle all of the keyword-manipulation now missing from Content Manager, EXCEPT managing the keywords from a GUI. My management scheme is still usable, just not convenient. Well, I`ll just have to find a way to work around that. trainzutil is a powerful beast, and I can wrap it in a Python-managed Tcl/Tk GUI of I need to. It will take some time, but it should be eminently doable. Theoretically, I could even recreate Content Manager itself this way. I may talk more about this later.

Update: still downloading, 4828MB/7904MB and counting, 590 minutes remaining.

I`m going to need to look into the possibility of downloading Trainz Content window items this way.

Long time ago in High School, I learned Basic and looked Machine based Languages Cobal and something else, Fortran?

I knew I could play with Basic, but those others way out of my Wheelhouse.

I enjoy your programing discussions, keeps my old Cobwebbed Brain enabled. Thank you for sharing about:

Wiki page for trainzutil.exe, and have learned that trainzutil can handle all of the keyword-manipulation now missing from Content Manager,

I opened this program a few times in the Past, them my OCD moved me onto other things. Looks like I need to revisit this Program.

 

[MasterTracklayer]

blueodesesy, thank you for liking my two earlier posts, however some of the information in one of them turned out to be inaccurate: Of the five asset directories I had in my Open for Edits directory last time I had looked, only three are still there. (Did I *move* them somewhere and forget?) Of those three, only one is at all usable. While waiting for my endless download to complete, I turned a copy of that one into something vaguely resembling the Hello World scripting tutorial on the Wiki. I don`t wish to jiggle the metaphorical arm of the downloading process, so I`m not going to test it until *much* later. Regarding the sharing of that information, I`m glad that it could entertain AND inform you at the same time. As I have said several times, in this and other threads, I can handle the impossible fairly easily (most of the time); it`s the easy stuff that throws me for loops, because it has been so long since I last used it. Actually, I think that most of my LIFE has been somewhat like that. :|

Edit: Status update: the self-restarted download now reports as 988MB of 7904MG, 994 minutes remaining. Great googly-moogly, it was *far* past this before it spontaneously restarted.

Edit 2: "Did I *move* them somewhere and forget?" Yes, I did; they are on my desktop, and the ones that I can understand even look usable!

 

[martinvk]

Yeah, the 700K+ is referring to the DLS itself - or is it? Mine shows "192946 of 710193 items displayed". So, the other 500K+ must be on the DLS. That being said, the 192946 displayed includes all of my base, bult-in, modified and third-party stuff, as well as packaged and payware, so what does that 700K+ number really represent?

In CM from Trainz22PE, build 123801, I show 77408 or 707977 items displayed. The 77408 is what is installed, including a few 3rd party items and a bunch of locally created things I made. I assume the larger 707977 is everything because when I filter for "All Content", that's the number I see. But when I filter for "Download Station" I get 428358 of 707977. Don't know why?

 

[MasterTracklayer]

My best guess, martinvk, is that you posses 279619 assets installed that did NOT originate on the DLS; is this possible? It would include built-in and packaged items, which may or may not also be available via the DLS, and it would also include all of the third-party assets you have, assuming that *they* are also not on the DLS. It will also include all of your personal content, like homemade routes, homemade sessions, home-made consists, all of which the game automatically creates assets for, and would also not be on the DLS, unless someone (you, most likely) has put them there.

Update: Currently 1354MB of 7904MG, 953 minutes remaining.

 

[martinvk]

About managing all those assets. My first question would be why have soooooo many in the CM? Unless you plan on modelling every type of railway from every location, it seems a bit of over kill. I find that having too many just dilutes the choices.
Every so often I will sort in installed date order and find all old items that have no dependants - ie they are not used in any of my routes. So if I haven't used it since downloading, it is a prime candidate for deletion.
I also have several picklists packed with frequently used items sorted by categories. Too many in one list? - perhaps a new category is needed.

 

[MasterTracklayer]

martinvk: all of your installed assets show up in your Content Manager, as well as those on the DLS. I have very *little* content installed yet, because this is a new installation (I first got Trainz22 at around Christmastime), and the content that I had accumulated with my Trainz Simulator 2 Pack all gotten eaten when the Windows XP on another computer decided to EAT my data partition. As for the over 20000 files referenced in my first post here, those are Windows filesystem objects (things like files and directories), NOT Trainz assets. I only had a bit under 6000 DLS assets installed in addition to the Packaged Content stored there. Packaged content, like built-in content, is stored in something more akin to a zip file than to a collection of separate filesystem entities. 20000+ filesystem entities for 6000- assets? Hmm, sounds typical, although I hadn`t realized at the time I had posted that I had already accumulated around 5500 DLS assets.

Status update: 1545MB of 7904MG, 935 minutes remaining.

 

[MasterTracklayer]

martinvk: I just thought of something to add in response to "why so many". Of those 5500 or so downloaded assets (and I think most people would agree that 5500 is an exceedingly SMALL collection), a bunch of them are buildings and/or industries, and a sprinkling of other asset types are there as well, but the vast bulk of them are rolling stock. What good are trains without train vehicles? (For that matter, what ARE trains without train vehicles?) I like some variety in my rolling stock. I`m even planning on building a tool to spawn randomly-generated consists onto routes (in both Surveyor and Driver modes). This tool will not needed to be updated when I add more rolling stock to my trainzbase; it will autodetect them and just start using them. It will, of course, have to be smart enough to detect what sort of traincar any given piece of rolling stock is, and at least *try* to use them "sensibly". For cases where it cannot deduce the correct usage of a vehicle, I`ll have to include a way to give the tool hints without modifying the tool itself.

Status update: 5170MB of 7904MG, 543 minutes remaining.

 

[martinvk]

Interesting but even so, you would have to stay within a narrow range of patterns. While it is possible that somebody imports a Euro-style Falns hopper, it would look a bit out of place in an North American train. Plus the different methods of coupling have to be considered. Of course Trainz has a universal coupler so anything can couple to anything. Heck, even differently gauged bogies can run on any gauge track, Trainz is that flexible.

So unless you model trains from around the world, you could easily delete other types and never miss them.

 

[JCitron]

Sorry, JCitron, but you misunderstand. I do not have a Win7 MACHINE, I *had* a Win7 Trainz installation on what turned out to be a Win10 machine. Both databases coexist in complete ignorance of each other in the standard locations on *one* computer. My Win10 Trainz22 is quite content using the database from the Win7 Trainz that **used to exist** on the same machine, but there are issues that *it* is blissfully unaware of.

If I use the Win7 database, I cannot download any "built-in"/"packaged" content that is different from the one build to the other. Is there any such content? I`d be surprised if there isn`t. If I use my Win10 database, I`ll have to reinstall everything downloaded with the Win7 install instead. Not a big issue, just a tedious one.

I also do not have an external hard drive available for the laptop, although I could use a USB thumb drive, if it`s big enough.

Any other suggestions? I`ll probably have to shoot them down, due to my oddball configuration (par for the course, with me; I always seem to manage to do something unexpected enough to make the Standard Solutions not work), but I`ll never know for sure if we don`t try.

You should be able to copy the packages folder from one version to the other. Once copied over, run a database repair to integrate the DLC and built-in content into the database. This will save you from having to download the DLC all over again. Since you already own it, there's no issues with doing it this way. We've been doing this since the DLC packages came out and it saves a ton of time.

Don't ever use a thumb drive! They are way too slow and are no more reliable than a floppy disc. I highly recommend a desktop-type backup drive such as Backups Plus or equivalent from Seagate. Don't get the kind that are powered only by the USB ports. Those are okay for reading and occasional writing such as for backing up content, but not for extensive reads and writes that Trainz does. In the olden days, well not that long ago, I used to do this when I traveled because my laptop has snot for disk space. I'd sit in the hotel room with my external drive plugged into a USB port and I'd Trainz away all evening. When I got home, I'd transfer any updated routes I had over to my desktop by plugging the drive into my desktop computer.