Written this way, you'll need roughly 6 bytes of memory for every character in the file.
Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.
If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.
You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.
Answer from erickson on Stack OverflowWritten this way, you'll need roughly 6 bytes of memory for every character in the file.
Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.
If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.
You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.
If your files to be processed are all very large, say more than a few hundred MB, then you really should go with stream processing instead of this "loading all into memory" way, just as @erickson suggested.
Otherwise, there are a few things you could try, all to reduce memory usage as much as possible:
- Try properly enlarge your heap size if not yet (when applicable).
- Give
StringBufferan initial size same as the lenght of the givenStringbuffer. This should reduce the unnecessary memory usage while expanding theStringBufferin the process. I assume it is only replacing certain words of the original string and should be more or less the same in length. - If possible, maybe you could return the generated
StringBufferobject instead. Calling itstoString()only after you get rid of the originalStringobject.
[BANano] [SOLVED] out of Memory heap ... at java.lang.StringBuilder.toString(StringBuilder.java:407)
OutOfMemory errors caused by debug logging of incoming Kafka messages
StringBuffer is dangerous when dealing with lonnnng Strings!!!
Out of memory exception on Stringbuilder
Hi All, bit of a novice here.
I'm trying to extract xml from a large string which contains mixed characters (code below). So it contains #### and others so I can't simply loadxml I have to extract the specific xml. I can use indexof to extract a request and a response xml which is what I want, but when I read through the whole string I run out of memory on the line stringBuilder.AppendLine(text);
Can anyone tell if there is a better way to get what I want without excessive memory?
Thanks in advance for your help.
public string AutoFormat(string result, string startTag)
{
string text;
var stringBuilder = new StringBuilder();
foreach (var index in result)
{
if (result.Contains($"<{startTag} ", StringComparison.OrdinalIgnoreCase) || result.Contains($"</{startTag}>", StringComparison.OrdinalIgnoreCase) || result.Contains(@$"<{startTag}Response xmlns=""http://www.tourenserver.de/""", StringComparison.OrdinalIgnoreCase) || result.Contains($"</{startTag}Response", StringComparison.OrdinalIgnoreCase))
{
int num = result.IndexOf($"<{startTag} ", StringComparison.OrdinalIgnoreCase);
int num2 = result.IndexOf($"</{startTag}>", StringComparison.OrdinalIgnoreCase);
int num1 = result.IndexOf(@$"<{startTag}Response xmlns=""http://www.tourenserver.de/""", StringComparison.OrdinalIgnoreCase);
int num3 = result.IndexOf($"</{startTag}Response", StringComparison.OrdinalIgnoreCase);
text = result.Substring(num, num2 - num + 7) + Environment.NewLine + Environment.NewLine + result.Substring(num1, num3 - num1 + 15);
AddLine(stringBuilder, text);
}
}
return stringBuilder.ToString();
void AddLine(StringBuilder stringBuilder, string text)
{
stringBuilder.AppendLine(text);
}
}Apparently your application doen't have enough memory for complete the operation. So you need to specify memory flags to your virtual machine. You can try the following:
java -Xms256m -Xmx512m YourApp
Where:
- Xms minimun memory allocated by your program at startup (in the example 256 MB)
- Xmx maximun memory allocated by your program if it need more (in the example 512 MB)
Well one of the things that might happen is the fact that in Java, java.lang.String gets special treatment. Strings are immutable, and thus the JVM places each String object in a pool .The role of this pool, amongs others is the fact that is used as a sort of "cache" where if you create multiple String instances having the actual "text" value identical, the same instance will be reused from the pool. This way creating what seems to be a big number of String object instances with the same text inside will infact revert to having very few actual String instances in memory. On the other hand, if you use the same text to initialize multiple StringBuilder instances, those will actually be separate instances (containing the same text), and thus occupying more memory.
On the other hand, concatenating Strings (such as String c = "a"+"b";) will in fact create more object instances that if you do it with StribgBuilder (such as new StringBuilder("a").append("b");).
Without knowing more about your program, I'll refrain from making any comments about the approach you've taken and comment only on the code you've posted. It sounds weird to me that a 44MB file is filling a 2G heap, though.
One thing you can do is allocate space in your target array up front:
ArrayList<String> convBuf = new ArrayList<String>(buf.size());
This will avoid the ArrayList resize step, which creates a copy of the existing list (and shows up in your stack trace).
Another thing you can try is to release references to the original StringBuilders while building the String array. Use an Iterator (instead of a "for each" loop), and remove the StringBuilder from the buff array using Iterator.remove() on each iteration; that way you'll free some memory for the garbage collector to reclaim if you're running out of memory.
But again, it sounds weird that you're running out of memory with such a small file. Maybe looking at your heap with jvisualvm can shed some light.
There are two ways to get value from StringBuilder
- First use
Reflectionand callpackage protectedmethod getValue() It is hard and I do not think that is so good. - Second call method getChars
BTW, you will have arrays of chars, and for use string you will create new instances of string, and it will use your memory.
What is the reason for the OOM while creating a new string
Because you're running out of memory - or at least, the CLR can't allocate an object with the size you've requested. It's really that simple. If you want to avoid the errors, don't try to create strings that don't fit into memory. Note that even if you have a lot of memory, and even if you're running a 64-bit CLR, there are limits to the size of objects that can be created.
and why it doesn't throw OOM while writing to a file ?
Because you have more disk space than memory.
I'm pretty sure the code isn't exactly as you're describing though. This line would fail to compile:
sw.write(SB.ToString());
... because the method is Write rather than write. And if you're actually calling SB.ToString(), then that's just as likely to fail as str = SB.ToString().
It seems more likely that you're actually writing to the file in a streaming fashion, e.g.
using (var writer = File.CreateText(...))
{
for (int i = 0; i < 5000; i++)
{
writer.Write(mytext);
}
}
That way you never need to have huge amounts of text in memory - it just writes it to disk as it goes, possibly with some buffering, but not enough to cause memory issues.
Workaround: Suppose you would want to write a big string stored in StringBuilder to a StreamWriter, I would do a write this way to avoid SB.ToString's OOM exception. But if your OOM exception is due to StringBuilder's content add itself, you should work on that.
public const int CHUNK_STRING_LENGTH = 30000;
while (SB.Length > CHUNK_STRING_LENGTH )
{
sw.Write(SB.ToString(0, CHUNK_STRING_LENGTH ));
SB.Remove(0, CHUNK_STRING_LENGTH );
}
sw.Write(SB);