Asset Database Backup: Wow! Over 20,000 files!

[MasterTracklayer]

I downloaded and installed Trainz22 for Win7 because I wasn`t sure what version of Windows I`m running. I then went on a minor downloading spree with Content Manager. Upon learning about "winver", I discovered that I have Win10, so I downloaded the current build. Each took HOURS to download with my slow slightly-flakey connection. It finally finished downloading, so I decided to make a simple-minded file-copy back-up of my installed Trainz data.

Windows reports that the source directory contains 20,432 items and will take "More than 1 day" to copy.

That is a LOT of VERY SMALL files!

...and I don`t even have a whole lot of content installed. Almost none compared to some of the reports in the Missing KUID thread, and I had downloaded a lot more from the DLS when I was using earlier versions on another computer, but still a drop in the bucket compared to the people I`ve encountered reading that other thread.

How many files in your Trainz data directory, and how do you manage all that content? The features that I relied on heavily in my earlier installs (managing asset keywords in Content Manager without editing the assets) do not exist in Trainz22, and trying to simulate them with Pick Lists seems likely to be extremely unwieldly. Ideas (and content-folder file-counts) anyone?

Edit: After settling down a bit, the file-copy window now reports "About 2 hours", but I am well aware of how that number can fluctuate drastically as the copy process encounters files of wildly-differing sizes. I`ve developed an algorithm that should account for these differences and report more-stable estimated-time-remaining values, but that doesn`t help anyone when I cannot install it in place of the unstable-estimate generators.

Edit 2: Wouldn`t ya know it? While the copy was in progress with around one hour and 20 or 30 minutes, our power flickered /very/ briefly. Now the machine I`m running on is an old laptop with (I think) about 20-30 minutes of operation on a fully-charged battery, so that shouldn`t bother the computer, right? WRONG! The computer flipped a reboot without clean shutdown. Now I`ve gotta clean up the residuum of the attempted copy and try again. Luckily, the *only* other thing I as doing at the time was browsing the messages here, and Google Chrome helpfully offered to re-open all the pages that I had open at the time.

 

[davesnow]

I have 710,621 files.

 

[MasterTracklayer]

Wow! I got replied to by the crabby old geezer himself! I`ve had filters featuring your author identifier in every Trainz installation Content Manager I can remember, and I`ve been around for a couple decades, albeit quietly. You do some pretty darn good stuff! Don`t worry, I use those filters to find more of your stuff to download!

DD Form 214? That couldn`t be a Dishonorable Discharge, could it? I`m no military expert, but that`s the only military DD I can think of.

...or perhaps that`s just a DoD form, possibly an honorable discharge? It didn`t occur to me at first that DoD might also be abbreviated DD.

P.S.: Do note Edit and Edit 2 in my initial post above.

 

[blueodessey]

I downloaded and installed Trainz22 for Win7 because I wasn`t sure what version of Windows I`m running. I then went on a minor downloading spree with Content Manager. Upon learning about "winver", I discovered that I have Win10, so I downloaded the current build. Each took HOURS to download with my slow slightly-flakey connection. It finally finished downloading, so I decided to make a simple-minded file-copy back-up of my installed Trainz data.

Windows reports that the source directory contains 20,432 items and will take "More than 1 day" to copy.

That is a LOT of VERY SMALL files!

...and I don`t even have a whole lot of content installed. Almost none compared to some of the reports in the Missing KUID thread, and I had downloaded a lot more from the DLS when I was using earlier versions on another computer, but still a drop in the bucket compared to the people I`ve encountered reading that other thread.

How many files in your Trainz data directory, and how do you manage all that content? The features that I relied on heavily in my earlier installs (managing asset keywords in Content Manager without editing the assets) do not exist in Trainz22, and trying to simulate them with Pick Lists seems likely to be extremely unwieldly. Ideas (and content-folder file-counts) anyone?

Edit: After settling down a bit, the file-copy window now reports "About 2 hours", but I am well aware of how that number can fluctuate drastically as the copy process encounters files of wildly-differing sizes. I`ve developed an algorithm that should account for these differences and report more-stable estimated-time-remaining values, but that doesn`t help anyone when I cannot install it in place of the unstable-estimate generators.

Edit 2: Wouldn`t ya know it? While the copy was in progress with around one hour and 20 or 30 minutes, our power flickered /very/ briefly. Now the machine I`m running on is an old laptop with (I think) about 20-30 minutes of operation on a fully-charged battery, so that shouldn`t bother the computer, right? WRONG! The computer flipped a reboot without clean shutdown. Now I`ve gotta clean up the residuum of the attempted copy and try again. Luckily, the *only* other thing I as doing at the time was browsing the messages here, and Google Chrome helpfully offered to re-open all the pages that I had open at the time.

Just for info, when computers do download or file install Copying etc, Large files do take some time, however tons of Small Files always take an inordinate amount of time to Copy Move Etc,

I think what slows the CPU down, the actual writing of new Files to each sector of your Hard Drive. I have played with Computers since Tandy T-1000 Vintage 1970, a very long time ago, even then, we were plagued with this issue.

Sad Commentary, Even SSD's are only so fast, depending on Bus Speed! I/O operations Etc.

However I would offer one point of advice, when doing huge copy commands, if at all possible close all non essential Programs, especially your Email and go into Airplane Mode, that way most of extraneous baloney is shutdown, and your CPU etc can give more dedication to Copy? Install command.

Also if you have any Backup Programs which run in the Background, shut them down or Pause them. It will make a difference over all.

 

[butler57]

Mines bigger than yours!
After some 18 years of asset collection from a range of sources my "Master Trainz 19 Content Manager shows 778112 items. However, the actual number of files will be way in excess of this - possibly millions. And over 1.9 terabytes.

 

[butler57]

As for The DD Form 214,
Certificate of Release or Discharge from Active Duty, generally referred to as a "DD 214", is a document of the United States Department of Defense, issued upon a military service member's retirement, separation, or discharge.

 

[MasterTracklayer]

blueodessy: Have you seen the edits to my original post? The algorithm mentioned in the first edit is designed especially to cope with this exact issue: per-file overhead is nearly constant while data transfer is usually a linear function of the size of the file. Conventional algorithms make no distinction and thus overreact to changes in file size. Mine considers them separately and issues estimates accordingly. Unfortunately, nobody uses my algorithm. I`ve been playing with computers for about that long too.

As for Airplane Mode, I`ve got that on my phone, but have never seen it in Windows. Regarding your other suggestions, I agree entirely, in principle, but I haven`t a clue how to do that without killing Windows 10. As it happens, it only took approximately two hours to copy it all, which is nothing compared to, oh, about twelve to fourteen or so hours to download Trainz22. Twice: Remember that I was uncertain which Windows I am running until I saw a post that mentioned winver, so I downloaded the Win7-compatable version, just to be safe, then downloaded the current one once I knew my OS was compatible.

Butler57: Thank you; that was (obviously -- see above) my second thought regarding the meaning of the reference.

Everybody: How do you manage everything? Like I said earlier, I made heavy use of editing custom keywords directly in Content Manager without editing the assets themselves. No longer possible. Simulating it with Pick Lists, while possible, looks likely to be unwieldly. I would filter on my custom keywords in such a way that miscategorized assets would appear in the correct category filters and not in thee wrong ones. I used this to make a finer-grained filter than Content Manager was able to handle without it. Unfortunately, this took LOTS of custom filters, and while running them was fast enough, editing them has always been, and still is, glacially slow. -- and the number of "items" that Windows counted included all the files and all the directories, all counted up into a single number. Windows has no idea of an item as Trainz counts them.

Edit: with the updated Trainz22 downloaded and my meagre asset collection backed up, I`m about to try installing the new version. Wish me luck, people!

Edit 2: Corrected an offensive spelling of a brand name.

 

[davesnow]

As for The DD Form 214,
Certificate of Release or Discharge from Active Duty, generally referred to as a "DD 214", is a document of the United States Department of Defense, issued upon a military service member's retirement, separation, or discharge.

Commonly referred to as a HONORABLE discharge.

 

[MasterTracklayer]

Hey, Geezer, like I said above, my very second idea. <wink> <thumbs-up>

Meanwhile, extracting my new download is taking twelve minutes, and should be just about done.

 

[JCitron]

I have about 250K files, or about 780 GB of content. This is down from the 2.5 TB I had. Like Butler, 20-years of content downloading eats a lot of disk space. As you saw in the other thread, I had a sudden purge of content and to be honest Trainz is a lot happier for it and so am I.

 

[MasterTracklayer]

Yeah, I did see that.

Meanwhile, the installer finished extracting whilst I was busy in the forums, and I`m ready to install the bugger. Any Famous Last Words of Advice you`d care to post while I look around a bit more?

...and how the bloody blue blazes (pardon my British) do you keep all those bits ORGANIZED? As yet, I don`t have enough content to need to put my Trainz22 on a diet, and when I do, I`ll need ways to know what the bleep THAT oddly-named and badly malcatagorized thing is. This *is* the part of my original post that no-one has addressed yet, ya know.

Edit: I`ve started the install. There sure is a lot of payload data to check!

Edit 2: Installing...

Edit 3: Operation complete, Now to see what, if anything, it trashed.

Edit 4: Had to re-set my graphics settings, as predicted elsewhere. Took a bit to enter and validate my identity, the Trainz Content window looks like I will have to redownload all of my "included with purchase" downloadables. Big surprise there. NOT.

Edit 5: My computer replaced this page with the Help Desk in the middle of editing. Fortunately I didn`t lose much. Anyway, installed the Win10 version over top of the Win7 version. Didn`t touch my custom content. Bleep, it didn`t even LOOK at my custom content. All still there, but inaccessible. Any easy way to import the downloads into the new version without the old version available? I suppose I could reinstall that in a different location and go back and forth, but there ought to be an easier way. My own personal assets are now effectively missing, all three or four of them. Big whoop. Nothing there worth the bother, and everything else is on DLS, if only I could identify it. I`ve poste a list of my BROKEN downloads here, but its mostly everything else I want. <SIGH> Meanwhile, still interested in your content management methods (and data directory sizes).

 

[JCitron]

The issue with content being weirdly named is something we've talked about here for ages. If we could rename assets easily, meaning without opening up the config.txt and editing them, it would go far but that's not how it works. The only way you can really organize content, such as by type, region, or however way you want, is to use pick-lists. Take your most used content by a particular author and create a pick-list, or as many of us do, create a pick-list of particular asset types or for a particular purpose.

Your pick-lists will now be available in Surveyor. Press CTRL-SHIFT+F to bring up the search filter, as opposed to CTRL+F for global search, and you'll see your pick-lists. You can also save filters here as well but there's a limited number before important parts of the interface disappear off the screen and you can't access those items due to the interface not scrolling.

 

[MasterTracklayer]

Yup. I can do that, or I can simulate the keyword changing feature using pick-lists in a different way. Unfortunately, both ways look cumbersome. My idea for simulating keyword-editing with them especially so, albeit more flexible. Rhetorical question: Why does N3V have to keep taking away nifty Auran features? Rhetorical answer: Because they can? Meanwhile, please note the edits in my previous post. We kinda cross-posted.

Anyway, I saw a post SOMEwhere by SOMEone in one of the areas that mere mortals probably can`t post in that hinted at the possible existence of an external tool to maintain those keyword edits the way the old Auran Content Manager did. I probably misunderstood or misremembered the post, but that sure would be handy, especially if they showed up in-game. If ya don`t mind having to rebuild the database with an external tool every time Trainz rebuilds it, there is probably a way to do it. I kinda doubt such a program exists, though. I could write it myself, if I had the slightest notion of how the database is stored. Of course I have no clue on that. <SIGH>

Edit: Of COURSE all my filters are missing as well. What happens if I just TELL the Win10 version to use the Win7 version`s storage location? With everything already backed up, all I have to lose is two hours of file-copy Restore operation, and that only if I want to restore anything the experiment messes with. I probably shoulda installed the new version somewhere other than on top of the old one. Live and learn. Mebbe someone else can learn from my misadventure.

Edit 2: At least the windows still open where and how big they were when I used the old ver.

 

[JCitron]

The old keywords, while useful, weren't very stable. An extended database repair would remove them, a more often than not program crash would do the same. When the "new" Content Manager came out with T: ANE, that and many other useful features disappeared.

If you ever sign up for Plus or purchase TRS22 Platinum Edition (PE), you can rename assets in Surveyor 2.0 by modifying the assets properties in the details panel. That's useful but it's too bad it's not available in Content Manager too. Being able to rename assets to something meaningful, especially for those really, really useful but virtually unknown assets due to their weird names, is really useful. Names can be changed so that track-objects can be prefixed or suffixed with something so we know what is what, even by region if we want to go that far.

The database is an SQL-Lite database but it's encrypted which means we can't do anything with it. I was hoping to be able to access the database in a read-only mode to be able to build my own queries. Being able to sort by assets as dependencies, for example, would be really useful especially when cleaning up content.

 

[MasterTracklayer]

VERY interesting, but VERY unhelpful. Not YOUR fault, of course. As for the limits of the method I suggested: exactly what I would expect for unofficialware. I even stated as much in the sentence that starts "If ya don`t mind having to rebuild the database with an external tool every time Trainz rebuilds it". Wink.

Those features of the new Surveyor sound great, but I fear my second-hand laptop won`t be powerful enough. Bleep, I expect it to be underpowered for thee version I HAVE. Not to mention can`t afford it. This was purchased as a Christmas-present splurge for myself.

I`m about to try pointing my new install at my old install`s data. Stay tuned to see how badly it messes up. Wink.

 

[PhilChorusch01]

I have 87,628 files

 

[Forester1]

Yeah, the 700K+ is referring to the DLS itself - or is it? Mine shows "192946 of 710193 items displayed". So, the other 500K+ must be on the DLS. That being said, the 192946 displayed includes all of my base, bult-in, modified and third-party stuff, as well as packaged and payware, so what does that 700K+ number really represent?

 

[MasterTracklayer]

Update: it looks like Win10 Trainz22 likes my Win7 Trainz22 data just fine. Every censored item appears to be right where I left it,

EXCEPT for a large number of assets showing as "Newer version available". What do people think: Should I just download updates with Content Manager, or try to export non-built-in assets and re-import them into the new install`s database? I`ll HAVE to do something like that if I want to migrate my own measly four whole assets over, but some of them aren`t worth the bother.

Alternatively, since every asset that isn`t mine came from the DLS, I could just filter on those assets, copy their kuids, and redownload.

The only work I did that I really want to preserve but can`t figure a way to migrate is a whole boatload of custom filters, which would take hours, perhaps a day or two, to recreate, especially with Trainz`s track record with excruciatingly slow operations on the filters themselves. Does anyone know how to migrate custom filters?

In any case, my one-month First Class Ticket expires in days, so I kinda gotta hustle here.

Edit: Just tried filtering on Installed True, Built-In False, On Download Station True in an attempt to identify my downloads for possible re-download and got 15818 hits, 5625 of which list as On Download Station (no additional qualifiers), with an additional 9 likewise with Faulty (all Auran trees), another additional with Missing Dependencies instead of Faulty, one that lists with Newer Version Available, and three that list as Obsolete. Remember that *all* of these include "Installed from DLS". One lists as "Incompatible". The remainder all list as "Packaged", many with "Payware" as well.


5625 kuids are too many to export in a single operation (see exporting picklists to .txt files), so I`ll have to do it in batches. Fun, Not.

 

[JCitron]

Copy your complete database as-is from your Windows 7 to your Windows 10 machine. Once copied over, you can then connect to your Win 7 database without any issue and that will allow you to keep all your filters and you'll lose nothing.

Once copied over, you can then do the recommended updates. I always update my content and delete obsolete assets. Keeping the obsolete versions eats up disk space.

Pro-Tech tip:

Put your data-folder on another hard drive.
Rename the folder to something other than Build-something. I call mine Plus_Data and it's on my E: drive.
Link your program to the data in the Launcher - Trainz settings - Install-tab.

 

[MasterTracklayer]

Sorry, JCitron, but you misunderstand. I do not have a Win7 MACHINE, I *had* a Win7 Trainz installation on what turned out to be a Win10 machine. Both databases coexist in complete ignorance of each other in the standard locations on *one* computer. My Win10 Trainz22 is quite content using the database from the Win7 Trainz that **used to exist** on the same machine, but there are issues that *it* is blissfully unaware of.

If I use the Win7 database, I cannot download any "built-in"/"packaged" content that is different from the one build to the other. Is there any such content? I`d be surprised if there isn`t. If I use my Win10 database, I`ll have to reinstall everything downloaded with the Win7 install instead. Not a big issue, just a tedious one.

I also do not have an external hard drive available for the laptop, although I could use a USB thumb drive, if it`s big enough.

Any other suggestions? I`ll probably have to shoot them down, due to my oddball configuration (par for the course, with me; I always seem to manage to do something unexpected enough to make the Standard Solutions not work), but I`ll never know for sure if we don`t try.