Some interesting Windows 10 information regarding memory usage...

Hi Trainzers,

I've just been reading some very interesting information regarding Windows 10 and it's new memory management system, which may explain some of the memory related issues involving T:ANE and Windows 10 especially if the system has not got much memory.

It appears that instead of paging memory to disk, Windows 10 now compresses unused memory pages. Whilst this reduces the amount of memory used by processes, it increases the working memory set of the System process (ntoskrnl) which appears to cause increased memory usage by that process and if it does so for too long can cause Windows to start to end other processes including T:ANE.

It looks like there may be improvements in the later Insider builds according to http://blogs.windows.com/windowsexp...ncing-windows-10-insider-preview-build-10525/.

I'm interested to see if anyone else has encountered this compression behaviour yet in Windows 10 - I know I have.

Shane

shaneturner12 said:

Hi Trainzers,

I've just been reading some very interesting information regarding Windows 10 and it's new memory management system, which may explain some of the memory related issues involving T:ANE and Windows 10 especially if the system has not got much memory.

It appears that instead of paging memory to disk, Windows 10 now compresses unused memory pages. Whilst this reduces the amount of memory used by processes, it increases the working memory set of the System process (ntoskrnl) which appears to cause increased memory usage by that process and if it does so for too long can cause Windows to start to end other processes including T:ANE.

It looks like there may be improvements in the later Insider builds according to http://blogs.windows.com/windowsexp...ncing-windows-10-insider-preview-build-10525/.

I'm interested to see if anyone else has encountered this compression behaviour yet in Windows 10 - I know I have.

Shane

Click to expand...

I have as well. I ended up doubling my page file even with 64GB of physical RAM installed to keep the out of memory conditions from occurring while performing a database repair. Even with the doubled page file, the system consumes 85% of the RAM before the process completes.

I'll take a look at the article. Thanks for posting the link. I've stayed out of the Insider builds since I built my new system.

John

Makes sense. Is it the System process that's causing it as well on yours?

Shane

shaneturner12 said:

Makes sense. Is it the System process that's causing it as well on yours?

Shane

Click to expand...

No, the system process is fine - it barely goes up. T:ANE on the other hand eats up all the RAM like a balloon filling up with air.

John

That's something that as yet hasn't happened to me, although it sounds like some form of memory leak.

Shane

JCitron said:

No, the system process is fine - it barely goes up. T:ANE on the other hand eats up all the RAM like a balloon filling up with air.

Click to expand...

If you are concerned about memory usage then it's not memory usage you need to look at. It's hard page faults. If you are getting a significant number of hard page faults then you don't have enough RAM for the applications you are running. If you are not getting hard page faults then you have enough RAM. If Windows is doing its job properly then all your installed RAM will be used and you will have no hard page faults. That means that the application has been allocated as much memory as it requires, without forcing the system into paging. That's the optimum state.

High memory usage simply tells you that Windows is being very efficient in using all the resources available to it, even perhaps to the extent that it is holding onto old data on the chance it will be requested again in the future, and perhaps even preloading data from disk based on past disk access patterns. Hard page faults are what indicates that the applications have a requirement for more RAM than Windows can make available. They will always occur when new things happen, but that should be evident as one-time spikes. If there is persistent high levels of paging then there is a problem with insufficient memory and you should close unnecessary applications. The compression store in Windows 10 just means you can load more stuff before the system starts paging, but it's still the hard paging that tells you that you have insufficient RAM, not the amount of memory in use.

I understand that fully, Dan.

My system has 64GB of RAM installed, and I just upped my two hard-drive based page files to 65GB each, this is in addition to the system one on the smaller C: drive. I will see if this makes a difference when a long database repair occurs. The memory issue occurs during the final phase of the database repair when the faulty assets are being checked. If T:ANE eats up my RAM now then there's definitely a problem with the program because 144GB of swap space should be adequate. In all my years as a network admin, my NT servers never had swap files that large.

John

JCitron said:

If T:ANE eats up my RAM now then there's definitely a problem with the program because 144GB of swap space should be adequate. In all my years as a network admin, my NT servers never had swap files that large.

Click to expand...

You are still confusing swap files with RAM. They are not the same. Swap files can easily grow to a huge size as Windows detects that some applications are idle, and optimizes RAM for the current high-priority jobs by swapping out everything that isn't currently being used. Windows also pre-allocates swap file space for possible swapping requirements that might never happen. The RAM usage you are seeing in a database repair is probably due to Windows assuming that recently accessed disk files are likely to be accessed again, and holding on to the sector buffers in RAM in preference to the inactive other applications. So it's not the size of the swap files that is important, any more than it is the amount of RAM in use. It's whether or not the current application (executable or data) is being swapped - and that's what the hard paging faults show.

If T:ANE 'eats up your RAM' then it's because Windows has determined that is the best usage for that RAM. It doesn't always get that right, of course, especially if you suddenly do something unexpected like start up your Resource Monitor. But it is not a problem - it is an indication that Windows is working hard to optimise the use of the available resources.

SailorDan said:

You are still confusing swap files with RAM. They are not the same. Swap files can easily grow to a huge size as Windows detects that some applications are idle, and optimizes RAM for the current high-priority jobs by swapping out everything that isn't currently being used. Windows also pre-allocates swap file space for possible swapping requirements that might never happen. The RAM usage you are seeing in a database repair is probably due to Windows assuming that recently accessed disk files are likely to be accessed again, and holding on to the sector buffers in RAM in preference to the inactive other applications. So it's not the size of the swap files that is important, any more than it is the amount of RAM in use. It's whether or not the current application (executable or data) is being swapped - and that's what the hard paging faults show.

If T:ANE 'eats up your RAM' then it's because Windows has determined that is the best usage for that RAM. It doesn't always get that right, of course, especially if you suddenly do something unexpected like start up your Resource Monitor. But it is not a problem - it is an indication that Windows is working hard to optimise the use of the available resources.

Click to expand...

Yes, I understand this and I am not confusing the two.

T:ANE will suck up every ounce of RAM when doing the faulty asset check. Seriously crash it. it's really that simple to do then do a database repair. When I say sucks up all the RAM, I mean it will cause Windows to see a low memory condition and want to shutdown applications. The applications, note plural in the Windows error message, is actually a single application called T:ANE. By increasing my page file settings, I was able to get around this condition and since then have not had this problem.

To add to this... Tonight I've been doing the usual in Surveyor including replacing assets which includes some textures. Prior to my current setup, T:ANE would crash to the desktop. Poof! Gone to bit heaven without warning. The only evidence is a crashdump.dmp file in T:ANE's user data directory. The subsequent result is then again a lengthy database repair and reentry of user information, configuration settings, and recreating custom filters in Content Manager.

So the increase in swap space apparently is working... This gives Windows a chance to move stuff out of the way to allow T:ANE to have the space it needs to operate.

John

JCitron said:

So the increase in swap space apparently is working... This gives Windows a chance to move stuff out of the way to allow T:ANE to have the space it needs to operate.

Click to expand...

Nearly. The correct description is : "So the increase in swap space apparently is working... This gives Windows a chance to manage the allocation of memory amongst the different tasks without crashing."

The application doesn't need that space - if it did, those 4Gb systems would never be able to do anything, let alone a DB repair. But it can use that space if it's there. But only if the information from the OS about the space being available is correct, and that will only happen if the swap file is big enough. You must size the swap file according to the RAM size - that has been the rule for every recent version of Windows. Equal to the RAM size is a good starting point, but double the RAM size is safer. Then, when Windows decides that more RAM should be allocated to the application, it will be able to allocate that RAM without crashing.

SailorDan said:

Nearly. The correct description is : "So the increase in swap space apparently is working... This gives Windows a chance to manage the allocation of memory amongst the different tasks without crashing."

The application doesn't need that space - if it did, those 4Gb systems would never be able to do anything, let alone a DB repair. But it can use that space if it's there. But only if the information from the OS about the space being available is correct, and that will only happen if the swap file is big enough. You must size the swap file according to the RAM size - that has been the rule for every recent version of Windows. Equal to the RAM size is a good starting point, but double the RAM size is safer. Then, when Windows decides that more RAM should be allocated to the application, it will be able to allocate that RAM without crashing.

Click to expand...

Yes...

I first used a system managed swap file, which for most systems works 99.9999% of the time and never had an issue with that. Trainz crashed during database repairs in HF2, and prior to that during submitting of lots of assets with a system out of memory message. I doubled the swap and the problem went away.

T:ANE is a 64-bit application so it should use available RAM, but for some reason it is not releasing unused memory in the process, thus, maybe causing the out of memory condition which is what we are seeing. What is more interesting and probably related to this is when the repair process finally finishes, the memory usage dropped down to 50% for T:ANE and just as Shane has confirmed the system ntosknl memory usage climbs substantially over time. The combination of the two brought the memory load to 96%.

My results are also echoed by Ian Woodmoore. In fact it was his observations which prompted me to doing the same as he. He has many, many years experience as a programmer and system engineer so I respect his findings. My experience only pales compared to his at 30 years.

That said, the only way to truly catch the memory performance of the application is to run Perfmon. The problem with running Perfom is it's another application running and its own load on the system can impact the performance. If necessary I will do this but I want to avoid this at this time for the obvious reason just stated.

Smaller problem

I am miles away from you guys in the complexity of my problem and in the complexity of your IT expertise. But. This thread seems to have elements related to what I am experiencing, so grateful for any help.

I have recently started using TS12, having finally bought a computer capable of using it. 8GB of RAM, Windows 8.1, as I have wanted to hear others' problems (if any) with Windows 10 before switching to that OS. TS12 repeatedly freezes, often with big patches of brown across the screen, and can be shut down only through task manager. I am building a small route in Surveyor and can sometimes do a couple of Driver/Surveyor iterations in Quick Drive before TS12 crashes. Other times it lets me do no more than a few moves around the route in Surveyor before the crash.

I am at build 49922 of TS12. The install fails if I try to go beyond that

JCitron said:

Yes...

I first used a system managed swap file, which for most systems works 99.9999% of the time and never had an issue with that. Trainz crashed during database repairs in HF2, and prior to that during submitting of lots of assets with a system out of memory message. I doubled the swap and the problem went away.

T:ANE is a 64-bit application so it should use available RAM, but for some reason it is not releasing unused memory in the process, thus, maybe causing the out of memory condition which is what we are seeing. What is more interesting and probably related to this is when the repair process finally finishes, the memory usage dropped down to 50% for T:ANE and just as Shane has confirmed the system ntosknl memory usage climbs substantially over time. The combination of the two brought the memory load to 96%.

My results are also echoed by Ian Woodmoore. In fact it was his observations which prompted me to doing the same as he. He has many, many years experience as a programmer and system engineer so I respect his findings. My experience only pales compared to his at 30 years.

That said, the only way to truly catch the memory performance of the application is to run Perfmon. The problem with running Perfom is it's another application running and its own load on the system can impact the performance. If necessary I will do this but I want to avoid this at this time for the obvious reason just stated.

Click to expand...

laldfordo said:

I am at build 49922 of TS12. The install fails if I try to go beyond that

Click to expand...

Fix that before you consider trying to run Trainz. You cannot just ignore a problem like that and expect other things to work properly.

Smaller problem

Thank you BuilderBob.

1. I tried changing from OpenGL to DirectX and, touch wood, that seems to have fixed the problem of repeated crashes.

2. I do not know how to fix the failure-to-patch problem. I get the same error message every time I try, namely, "Failed to patch Builtin\ts12c.ja, file contents do not match." I asked Auran, which said to keep trying. But that is just going round in circles. If the patch contains a new version of Builtin\ts12c.ja, why does it not just replace the file? Any ideas please?

BuilderBob said:

Fix that before you consider trying to run Trainz. You cannot just ignore a problem like that and expect other things to work properly.

Click to expand...