The Dangers of the Large Object Heap(转载,LOH内存碎片情景重现)
原文地址:https://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/
You'd have thought that memory leaks were a thing of the past now that we use .NET. True, but we can still hit problems. We can, for example, prevent memory from being recycled if we inadvertently hold references to objects that we are no longer using. However, there is another serious memory problem in .NET that can happen out of the blue, especially if you are using large object arrays. Andrew Hunter explains...
Usually, .NET developers don’t need to think too much about how their objects are being laid out in physical memory: after all, .NET has a garbage collector and so is capable of automatically removing ‘dead’ objects and rearranging memory so that the survivors are packed in the most efficient manner possible. The garbage collector has limits to what it can do, however; and when these limits are reached, then the runtime can exhaust memory in a way that is surprising and confusing to any developer who is not aware of how .NET chooses to lay out objects in memory.
.NET manages memory in four different regions, known as heaps. You can think of each of these as continuous regions of memory, though in practice .NET can create several fragmented regions for each heap. Three of the heaps, called the generation 0, 1 and 2 heaps are reserved for small objects: In current versions of .NET ‘small’ means objects that are 85,000 bytes or less. Any object is assigned to one of these generations according to the number of garbage collections it has survived, the veterans ending in generation 2. .NET knows that younger objects are more likely to be short lived and can reduce the performance cost of garbage collections by initially only looking at the recently allocated objects on generation 0. Perhaps more importantly, it can also move the survivors of a collection around so that there are no gaps, ensuring that the free space available for new objects is always together in one large lump. This helps with performance - .NET never needs to search for a hole big enough for a new object, unlike unmanaged code: if there’s enough memory available it’s always in the same place. When it needs to compact a series of objects, .NET actually copies all of them to a new block of memory rather than moving them in place; this improves performance by simplifying how objects are allocated. In these small heaps this means that the free space is always at the end, so there is never any need to scan elsewhere for a ‘hole’ big enough to store a new object.
The fourth heap is known as the Large Object Heap, or LOH. ‘Big’ objects go here - as the size at which an object may end up on this heap is 85,000 bytes, this usually means arrays with more than about 20,000 entries. It’s treated separately by the garbage collector, which will generally try to reclaim space from the other heaps before trying to tackle the giant objects that lurk here. Large objects pose a special problem for the runtime: they can’t be reliably moved by copying as they would require twice as much memory for garbage collection. Additionally, moving multi-megabyte objects around would cause the garbage collector to take an unreasonably long time to complete.
.NET solves these problems by simply never moving large objects around. After large objects are removed by the garbage collector, they leave behind holes in the large object heap, thereby causing the free space to become fragmented. When there’s no space at the end of the large object heap for an allocation, .NET searches these holes for a space, and expands the heap if none of the holes are large enough. This can become a problem. As a program allocates and releases blocks from the large object heap, the free blocks between the longer-lived blocks can become smaller. Over time, even a program that does not leak memory, and which never requires more than a fixed amount of memory to perform an operation, can fail with an OutOfMemoryException at the point that the largest free block shrinks to a point where it is too small for the program to use. In the worst cases, the amount of usable memory can shrink at a rate that seems unbelievable.
The worst case scenario
The design of the Large Object Heap means that its’ worse case will occur when the short-lived arrays are large and can vary in size a bit, and if the longer-lived arrays are small. The variation in size makes it likely that, when a new short-lived array is allocated, it won’t fit into the space left by one of its predecessors; .NET will then have to extend the LOH to ever-greater extents. This might not seem so bad: surely the smaller blocks can be put in the large gaps that are left behind?
Well, they can, but there’s a catch. As mentioned previously, .NET prefers to put new memory blocks at the end of the heap, as it has to do less work to find a space where a block will fit: This means that when there’s a new large block on the heap, the next smaller block will be allocated immediately after it, with the consequence that it will have to further extend the heap if the program ever wants to allocate a larger block.
What does this look like in a practical program? It seems fairly esoteric: allocate something big, then something small, keep the small thing and throw away the big thing. There are a few scenarios where this pattern can appear, and it can drastically reduce the amount of memory available to a program. The most trivial scenario could just be this:
List < int > list;
...
list.Add(x);
A List object’s underlying implementation is an array and when it needs to grow, .NET allocates a new, larger array and throws the old one away. All that’s needed to start to cause problems with fragmentation is to put some other object on the LOH before the array needs to grow. The main effect here is that you might be surprised how small the size is that your ‘large’ list can grow to before .NET tells you that it has run out of memory - long before the program is using up its 2Gb theoretical maximum.
#
Perhaps a more interesting case is this: a program is designed to read in a large data file, one that varies in size but is usually around 16Mb or so. It reads this into memory, processes it, and then produces a 90k summary. It is later adapted to run as a batch job: processing many of these files and storing the results in memory. Quite a lot of data processing applications can look a bit like this. You might expect that it could process tens of thousands of files before memory becomes a problem: after all only the 90k summary is kept between files, but testing surprises us with failure after only a few hundred files, by using up much more memory than expected.
The following program demonstrates this case. It fills up a list with 90,000 byte arrays and prints out how many bytes were allocated before .NET throws an OutOfMemoryException. It can also, optionally, allocate larger arrays between these allocations. These are short-lived, so they don’t look like they should significantly affect the number of smaller arrays that can be allocated. Unfortunately, each time the program allocates a larger array it allocates one more byte than last time.
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace LOH_test
{
class Program
{
/// <summary>
/// Static variable used to store our 'big' block. This ensures that the block is always up for garbage collection.
/// </summary>
static byte[] bigBlock;
/// <summary>
/// Allocates 90,000 byte blocks, optionally intersperced with larger blocks
/// </summary>
static void Fill(bool allocateBigBlocks, bool grow, bool alwaysGC)
{
// Number of bytes in a small block
// 90000 bytes, just above the limit for the LOH
const int blockSize = 90000;
// Number of bytes in a larger block: 16Mb initially
int largeBlockSize = 1 << 24;
// Number of small blocks allocated
int count = 0;
try
{
// We keep the 'small' blocks around
// (imagine an algorithm that allocates memory in chunks)
List<byte[]> smallBlocks = new List<byte[]>();
for (; ;)
{
// Write out some status information
if ((count % 1000) == 0)
{
Console.CursorLeft = 0;
Console.Write(new string(' ', 20));
Console.CursorLeft = 0;
Console.Write("{0}", count);
Console.CursorLeft = 0;
}
// Force a GC if necessaryry
if (alwaysGC) GC.Collect();
// Allocate a larger block if we're set up to do soso
if (allocateBigBlocks)
{
bigBlock = new byte[largeBlockSize];
}
// The next 'large' block will be just slightly largerer
if (grow) largeBlockSize++;
// Allocate a new block
smallBlocks.Add(new byte[blockSize]);
count++;
}
}
catch (OutOfMemoryException)
{
// Force a GC, which should empty the LOH again
bigBlock = null;
GC.Collect();
// Display the results for the amount of memory we managed to allocate
Console.WriteLine("{0}: {1}Mb allocated"
, (allocateBigBlocks ? "With large blocks" : "Only small blocks")
+ (alwaysGC?", frequent garbage collections":"")
+ (grow?"":", large blocks not growing")
, (count * blockSize) / (1024 * 1024));
}
}
static void Main(string[] args)
{
// Display results for cases both with and without the larger blocks
Fill(true, true, false);
Fill(true, true, true);
Fill(false, true, false);
Fill(true, false, false);
Console.ReadLine();
}
}
}
When run on a 32-bit Vista system, the program gives the following result:
Oh dear. When the program tries to allocate the larger blocks, it’s a disaster - it only manages to end up using around 20Mb of long-term storage, out of a theoretical maximum of 2GB! Forcing garbage collections barely helps at all. Even in the case where the larger blocks don’t grow the amount of memory the program can use is reduced by over half. If the large blocks are taken away, the program easily creates 1.6GB of data.
The last case in which the blocks don’t grow, but the amount of available memory is still reduced, is interesting. When the LOH fills up and the program wants to allocate a new small block, .NET tries to re-use free space before it grows the heap: this means that the space where the larger block was previously being put becomes too small, and the LOH has to grow in a similar way to before. It’s not as critical, but it does still result in the amount of available memory slowly going down.
Switching to a 64-bit system will improve matters: the program will be able to use up a lot more memory before it fails, but perhaps the situation is worse - because that memory is assigned to the program, the operating system will be paging it all out to disk resulting in considerably reduced performance for every program on the system and perhaps eventually running out of disk space.
It is difficult to diagnose this problem with the tools that are currently available. The best approach available at the moment is to combine a memory profiler with the performance counters. You can look at the ‘Large Object Heap Size’ performance counter to discover how many bytes are allocated to the large object heap. This counter includes free space, so to discover whether or not the problem is that memory has run out or because the memory that is available has become too fragmented to be used you will need to use a memory profiler. A good profiler will be able to tell you the total size of all of the live objects on the large object heap. If there is a big discrepancy between the two numbers, then it is possible that your program is suffering from heap fragmentation. Unfortunately, this isn’t guaranteed: there’s no problem if there’s a large amount of free space at the end of the heap - .NET just won’t clean it up until it has to for performance reasons.
Working around the problem
It might seem that nothing can be done to solve this problem apart from periodically stopping and restarting the afflicted program. This is probably the case if there is really no choice other than to deal with very large arrays of objects (or very large strings), but often it is possible to reduce or eliminate the need to use the large object heap, or to use it in such a way as to prevent fragmentation from occurring.
The best way to prevent fragmentation from occurring is to ensure that no objects above 85k in size are used for permanent storage. This ensures that the free space in the large object heap will always end up in one big chunk as short-lived objects are removed from play.
If large data structures need to live for a long time, and especially if they need to grow in size over time, the best approach is simply to consider using or writing a different data structure to store them. Arrays can contain up to around 10,000 elements before they are put on the large object heap and can cause problems, so a very effective way to store 100,000 entries might be to store 10 arrays each containing 10,000 elements: none will end up on the large object heap so no fragmentation will occur. This could be written as an IList subclass, which would make it easy to drop in transparently to replace existing code.
Depending on the design of the code, a simpler approach might simply be to make the ‘temporary’ but large structures more permanent. If a program needs to deal with a set of large files, it will make more efficient use of memory if it allocates enough space to deal with the largest one first, and then re-uses that for every file instead of allocating just enough for each and then throwing the memory away later. Many of the classes in System.Collections have a ‘Capacity’ property to facilitate this design pattern, and thread local static variables can be used (with care) to help share a large data structure in multithreaded applications.
It is unfortunate that the CLR garbage collector deals with large objects in this manner. The effect is similar to a memory leak, but one where we get no clue as to the cause by measuring the size of the objects in memory: a memory profiler will tell you that there’s no problem, while task manager tells you that the program’s memory usage is growing ever larger – which could be due to this issue or which could be due to the behavior of unmanaged code. What is needed instead is a way of finding out where objects are in memory and how they are affecting the free memory available to .NET. This can be done with some detective work: code can be inspected for areas where large amounts of memory are likely to be allocated, and a memory profiler can be used to help discover these areas. Once these are known, a debugger can be used to step through the code and some pen and paper can give an idea of what the large object heap looks like, which can be used to decide on the best way forward.
There is a feature on the drawing board for ANTS Memory Profiler 5that will help diagnose this issue and distinguish it from an unmanaged memory leak, and there are plans for features in future versions that will help isolate the specific cause once the issue is diagnosed.
The Dangers of the Large Object Heap(转载,LOH内存碎片情景重现)的更多相关文章
- 关于LOH(Large Object Heap)及内存泄漏
关于LOH(Large Object Heap)的. .NET CLR中对于大于85000字节的内存既不像引用类型那样分配到普通堆上,也不像值类型那样分配到栈上,而是分配到了一个特殊的称为LOH的内部 ...
- Large Object Heap内存碎片在.NET 4.5中的改进
.NET 4.5已然到来,预览了解了下Large Object Heap在.NET 4.5中的效能改进.借此和大家来探讨下.本文不讨论Loder Heap,SOH(samll object heap) ...
- ionic android - Unable to start the daemon process. Could not reserve enough space for 2097152KB object heap
Unzipping C:\Users\app\.gradle\wrapper\dists\gradle-4.1-all\bzyivzo6n839fup2jbap0tjew\gradle-4.1-all ...
- eclipse scala Could not reserve enough space for object heap
[学习笔记] 防盗版实名手机尾号:73203 Could not reserve enough space for object heap 解决方法:下面的缩写中:一个是memory max(Xmx) ...
- Android Studio: Failed to sync Gradle project 'xxx' Error:Unable to start the daemon process: could not reserve enough space for object heap.
创建项目的时候报错: Failed to sync Gradle project 'xxx' Error:Unable to start the daemon process: could not r ...
- Tomcat启动报错 Could not reserve enough space for object heap
报错信息: Error occurred during initialization of VM Could not reserve enough space for object heap Coul ...
- 启动tomcat报错 Could not reserve enough space for object heap的解决办法
问题:打开eclips启动tomcat发现报出Could not reserve enough space for object heap错误. 解决办法:1.首先检查tomcat是否能正常启动.re ...
- Error occurred during initialization of VM Could not reserve enough space for 2097152KB object heap
ionic build Android后的报错问题 ionic 升级了splashscreen和statusbar的插件后,执行ionic build android会一直报打包错误.原因是过低的An ...
- Error occurred during initialization of VM Could not reserve enough space for object heap
Error occurred during initialization of VM Could not reserve enough space for object heap Java虚拟机(JV ...
随机推荐
- 导出当前python安装了哪些第三方模块+批量安装python模块
pip freeze > mokuai.txt #导出你当前python环境里面有哪写第三方模块 pip install -r mokuai.txt #从文件里面批量安装模块
- VS2010设置默认浏览器
1.选择项目中起始页右键,然后选择浏览方式 2.然后就可以设置默认浏览器了
- 1. 时间复杂度(大O表示法)以及使用python实现栈
1.时间复杂度(大O表示法): O(1) < O(logn) < O(n) < O(nlogn) < O(n2) < O(n3) < O(2n) < O(n! ...
- Ntrip协议使用流程及服务器实现架构
Ntrip协议 在RTK接收机中,有多种方式获取到差分数据:电台.挂外串口.网络等.其中网络方式适合在有网络基准站的地区,省掉携带多一台RTK接收机来完成工作. CORS是网络基准站,通过网络收发GP ...
- 初用sqlite3.exe
1.记得要先建立数据库文件 为了进行数据库的编写,我安装了sqlite3,由于刚接触数据库,我尝试着建立表,并插入元组,属性,用select from语句也可以调出写入的内容,但是不知道如何保存,直接 ...
- Qt 学习之路 2(54):剪贴板
Qt 学习之路 2(54):剪贴板 豆子 2013年6月8日 Qt 学习之路 2 2条评论 剪贴板的操作经常和前面所说的拖放技术在一起使用.大家对剪贴板都很熟悉.我们可以简单地把它理解成一个数据存储池 ...
- 创建逻辑卷LVM以及swap分区
#!/bin/bash ##创建逻辑卷LVM /dev/mapper/lvm_data-data### ###default 大小为500G,但是LV一般会比500略小################ ...
- Day45--js基本小结
JavaScript基本总结 一:基本背景 01:注:ES6就是指ECMAScript 6.(2015 ECMAScript6 添加类和模块) ECMAScript和JavaScript的关系 199 ...
- POJ 2182 Lost Cows (求序列第k大)
题解 二分+树状数组 显然最和一个数的值就是rank 那么其它数有什么规律? 从后往前匹配rank,我们可以发现第i个数的rank为还没有匹配的rank第(a[i]+1)大的数 这可以用 树状数组+二 ...
- Luogu P1436 棋盘分割 暴力DP
我的天,,,,,n=8,k<=15,,,这怕不是暴力DP+高维数组.... 开一个五维数组f[k][i][j][p][q]表示从(i,j)到(p,q)中分成k个矩形最小的平方和. 然后初始化时用 ...