You should be running these tests in separate methods as the optimisation of the first loop can interfere with the optimisation of the second i.e. the second one can be slower, just because it is second.
I suggest you run both tests at least 10 (or for 2 seconds) and use System.nanoTime() which has higher resolution.
If you do this and you still run out of memory I suggest you increase the maximum memory size. If you are running 32-bit windows, the default is very low. You can increase it with -Xmx1g on the command line
If you run the following, you can see that GCs have the biggest impact which is not surprising as the problem most produces garbage
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
public class Collectionss {
public static final int TO_ADD = 10000000;
public static void main(String[] args) {
for (int i = 0; i < 5; i++) {
long timeAL = timeAddToArrayList();
long timeLL = timeAddsToLinkedList();
System.out.printf("Time to add %,d Integer to ArrayList %.3f sec, LinkedList %.3f%n",
TO_ADD, timeAL / 1e9, timeLL / 1e9);
}
}
private static long timeAddToArrayList() {
long starttime = System.nanoTime();
List<Integer> l1 = new ArrayList<Integer>();
for (int i = 1; i <= TO_ADD; i++) {
l1.add(i);
}
assert TO_ADD == l1.size();
return System.nanoTime() - starttime;
}
private static long timeAddsToLinkedList() {
long starttime = System.nanoTime();
List<Integer> l2 = new LinkedList<Integer>();
for (int i = 1; i <= TO_ADD; i++) {
l2.add(i);
}
assert TO_ADD == l2.size();
return System.nanoTime() - starttime;
}
}
prints
Time to add 10,000,000 Integer to ArrayList 0.238 sec, LinkedList 1.326
Time to add 10,000,000 Integer to ArrayList 1.193 sec, LinkedList 0.971
Time to add 10,000,000 Integer to ArrayList 0.841 sec, LinkedList 0.048
Time to add 10,000,000 Integer to ArrayList 0.349 sec, LinkedList 1.128
Time to add 10,000,000 Integer to ArrayList 0.064 sec, LinkedList 0.048
However add System.gc() before each test and you get
Time to add 10,000,000 Integer to ArrayList 0.241 sec, LinkedList 2.130
Time to add 10,000,000 Integer to ArrayList 0.070 sec, LinkedList 0.072
Time to add 10,000,000 Integer to ArrayList 0.067 sec, LinkedList 0.053
Time to add 10,000,000 Integer to ArrayList 0.069 sec, LinkedList 0.048
Time to add 10,000,000 Integer to ArrayList 0.065 sec, LinkedList 0.051
Answer from Peter Lawrey on Stack OverflowYou should be running these tests in separate methods as the optimisation of the first loop can interfere with the optimisation of the second i.e. the second one can be slower, just because it is second.
I suggest you run both tests at least 10 (or for 2 seconds) and use System.nanoTime() which has higher resolution.
If you do this and you still run out of memory I suggest you increase the maximum memory size. If you are running 32-bit windows, the default is very low. You can increase it with -Xmx1g on the command line
If you run the following, you can see that GCs have the biggest impact which is not surprising as the problem most produces garbage
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
public class Collectionss {
public static final int TO_ADD = 10000000;
public static void main(String[] args) {
for (int i = 0; i < 5; i++) {
long timeAL = timeAddToArrayList();
long timeLL = timeAddsToLinkedList();
System.out.printf("Time to add %,d Integer to ArrayList %.3f sec, LinkedList %.3f%n",
TO_ADD, timeAL / 1e9, timeLL / 1e9);
}
}
private static long timeAddToArrayList() {
long starttime = System.nanoTime();
List<Integer> l1 = new ArrayList<Integer>();
for (int i = 1; i <= TO_ADD; i++) {
l1.add(i);
}
assert TO_ADD == l1.size();
return System.nanoTime() - starttime;
}
private static long timeAddsToLinkedList() {
long starttime = System.nanoTime();
List<Integer> l2 = new LinkedList<Integer>();
for (int i = 1; i <= TO_ADD; i++) {
l2.add(i);
}
assert TO_ADD == l2.size();
return System.nanoTime() - starttime;
}
}
prints
Time to add 10,000,000 Integer to ArrayList 0.238 sec, LinkedList 1.326
Time to add 10,000,000 Integer to ArrayList 1.193 sec, LinkedList 0.971
Time to add 10,000,000 Integer to ArrayList 0.841 sec, LinkedList 0.048
Time to add 10,000,000 Integer to ArrayList 0.349 sec, LinkedList 1.128
Time to add 10,000,000 Integer to ArrayList 0.064 sec, LinkedList 0.048
However add System.gc() before each test and you get
Time to add 10,000,000 Integer to ArrayList 0.241 sec, LinkedList 2.130
Time to add 10,000,000 Integer to ArrayList 0.070 sec, LinkedList 0.072
Time to add 10,000,000 Integer to ArrayList 0.067 sec, LinkedList 0.053
Time to add 10,000,000 Integer to ArrayList 0.069 sec, LinkedList 0.048
Time to add 10,000,000 Integer to ArrayList 0.065 sec, LinkedList 0.051
If you are dealing with large data sets, then its always better to initialise your ArrayList with the correct size, in your case.
List<Integer> l1 = new ArrayList<Integer>(10000000);, otherwise your ArrayList will be having a default size of 10, and every time once the size is exceeded, the add method will create a new array of increased size copying all the contents of the old array inside your arraylist.
See the source below from ArrayList ensureCapacity method.
if (minCapacity > oldCapacity) {
Object oldData[] = elementData;
int newCapacity = (oldCapacity * 3)/2 + 1;
if (newCapacity < minCapacity)
newCapacity = minCapacity;
// minCapacity is usually close to size, so this is a win:
**elementData = Arrays.copyOf(elementData, newCapacity);**
}
And if its still happens, even after initialising then increase your heap size as others suggested in the post.
java - ArrayList Out of Memory Error - Stack Overflow
arrays - ArrayList and java.lang.OutOfMemoryError - Stack Overflow
java - Behaviour of JVM during out of memory error? List s = new ArrayList<String>(); - Stack Overflow
collections - ArrayList - java.lang.OutOfMemoryError: Java heap space - Stack Overflow
You never increment j. You are incrementing ii and val inside the for loop where as I think you actually want to do it in the while loop. Try the example below.
ArrayList<Integer> al = new ArrayList<Integer>();
Random rand = new Random();
int val=1;
int ii =0;
while(val < 400){
for (int j = 0; j<ii; j++) {
int pick = rand.nextInt(100);
al.add(pick);
}
ii++;
val++;
System.out.println("Contents of al: " + al);
al.clear();
System.out.print("Array has been cleared: " + al);
}
Your task would be much easier if you gave your variables meaningful names. e.g.
ArrayList<Integer> al = new ArrayList<Integer>();
Random rand = new Random();
int lineNumber = 1;
int lineLength = 0;
while(lineNumber < 400){
for (int randCount = 0; randCount < lineLength; randCount++) {
int pick = rand.nextInt(100);
al.add(pick);
}
lineLength++;
lineNumber++;
System.out.println("Contents of al: " + al);
al.clear();
System.out.print("Array has been cleared: " + al);
}
for (int j = 0; j<ii;) {
ii++;
val++;
pick = rand.nextInt(100);
al.add(pick);
}
is an infinite loop(j is always 0 and ii is always positive). You should fix it(according to the desired semantics).
Presumably the System.out.println call requires less memory than the s.add("Pradeep") call.
If s is an ArrayList for instance, the s.add call may cause the list to attempt to double up it's capacity. This is possibly a quite memory demanding operation, thus it is not very surprising that the JVM can continue executing even if it can't perform such relatively expensive task.
Here is more simple code that demonstrated better what happens:
try {
int[] a = new int[Integer.MAX_VALUE];
} catch( Exception e ) {
e.printStackTrace();
}
The allocation of the array fails but that doesn't mean Java has no free memory anymore. If you add items to a list, the list grows in jumps. At one time, the list will need more than half of the memory of the VM (about 64MB by default). The next add will try to allocate an array that is too big.
But that means the VM still has about 30MB unused heap left for other tasks.
If you want to get the VM into trouble, use a LinkedList because it grows linearly. When the last allocation fails, there will be only very little memory to handle the exception:
LinkedList<Integer> list = new LinkedList<Integer>();
try {
for(;;) {
list.add(0);
}
} catch( Exception e ) {
e.printStackTrace();
}
That program takes longer to terminate but it still terminates with an error. Maybe Java sets aside part of the heap for error handling or error handling doesn't need to allocate memory (allocation happens before the code is executed).
Hi, I'm doing some homework where I have to make a list of student objects and perform certain operations to them, and I need to have two classes, one using an Array and one using an ArrayList. My issue I'm having is I am inserting an object into the Arraylist at a specific index, but its throwing an error.
Here is my code:
public void insertStudent(String studentName, String newName, int q1, int q2, int q3, int q4, int q5) {
Student newStudent = new Student(newName, q1, q2, q3, q4, q5); //Creates the new student using the given values
for (int i = 0; i < myClass.size(); i++) //Goes through the list of students myClass
if (myClass.get(i).name.equals(studentName)) //Checks if the student has the name requested to replace
{
myClass.add(i, newStudent);
}and this is the error I'm getting:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3210) at java.util.Arrays.copyOf(Arrays.java:3181) at java.util.ArrayList.grow(ArrayList.java:261) at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235) at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227) at java.util.ArrayList.add(ArrayList.java:475) at TestStudent2.insertStudent(TestStudent2.java:42) at studentTester.main(studentTester.java:15) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
I'm honestly confused why I'm getting this, when I comment out
myClass.add(i, newStudent);
I don't get the error, but it looks to me like it should work. Any help would be appreciated.
For what it's worth this is the code I used in the Array portion of the assignment, it works fine.
public void insertStudent(String studentName, String newName, int q1, int q2, int q3, int q4, int q5) {
Student newStudent = new Student(newName, q1, q2, q3, q4, q5); //Creates the new student using the given values
Student[] tempClass = new Student[myClass.length + 1];
boolean t = false;
for (int i = 0; i < myClass.length; i++)
if (!myClass[i].name.equals(studentName) && !t)
tempClass[i] = myClass[i];
else if (myClass[i].name.equals(studentName)) {
t = true;
tempClass[i] = newStudent;
} else if (t)
tempClass[i] = myClass[i - 1];
tempClass[tempClass.length - 1] = myClass[myClass.length - 1];
myClass = tempClass;
}Also here are Pastebins for the entire project if you need some context.
Student.java
TestStudent.java (Using an Array)
TestStudent2.java (Using an ArrayList, the class with the issue)
studentTester.java (My tester class)
You need to increment your count
int count=0;
while (count<farray.size()){
FartsNameArray.add(farray.get(count).name);
count++;
}
You can also use a for loop too
for(int count = 0; count < farray.size(); count++) {
FartsNameArray.add(farray.get(count).name);
}
Or a foreach
foreach(Farts f in farray){
FartsNameArray.add(f.name);
}
i think you forgot to increment your count;
while (count<farray.size()){
FartsNameArray.add(farray.get(count).name);
count++;
}
You should update existing ArrayList elements with the set method. By adding to the ArrayList you are increasing its size so your loop never terminates since the end condition is for i to be equal to the size minus one. This condition is never met so your collection keeps growing until you run out of memory.
for(int i=0; i<al.size()-1; i++){
tmp = al.get(i+1);
al.set(i+1,al.get(i));
al.set(i, tmp) //assign the n+1 element to the n position
}
So the problem here is that you are adding to the arraylist, but you never take anything out. You have to remove the second value and then add it back in.
temp = al.remove(i+1);
or
temp = al.get(i+1);
al.remove(i+1);
Since you never delete that value, the arraylist grows bigger for every new value, like an exponential function.
Look at your loop:
while(j<campos.length){
nombres_reuniones.add(campos[j]);
}
How do you anticipate that ever finishing? You don't modify j. Given that you don't make any change to j after declaring it and assigning it a value of 0 right at the start, it would be much clearer as:
for (int j = 0; j < campos.length; j++) {
nombres_reuniones.add(campos[j]);
}
Or better:
for (String item : campos) {
nombres_reuniones.add(item);
}
Or even simpler:
nombres_reunions.addAll(Arrays.asList(campos));
Additionally, your earlier code can be simpler. Look at this:
String aux2 = null;
aux2 = aux.replace("[", "");
aux2= aux2.replace("]", "");
Why bother assigning aux2 an initial value of null which you then immediately overwrite? Additionally, you can easily chain the method calls. It would be neater as:
String aux2 = aux.replace("[", "").replace("]", "");
And in fact, you can just chain the whole of the string manipulation together from start to finish:
String[] campos = jsonReuniones.getString("nombres")
.replace("[", "")
.replace("]", "")
.split(",");
nombres_reunions.addAll(Arrays.asList(campos));
(I'd stop there, rather than inlining even that expression...)
You're not advancing the loop:
while (j < campos.length) {
nombres_reuniones.add(campos[j]);
j++; // without this, you'll loop forever!
}
What's happening is that you're adding an infinite amount of "campos" to the ArrayList, exhausting all the memory available to your program in the process.
Remember: a loop's condition must be false at some point for the loop to end. If you forget to advance the loop (in this case, by incrementing the j variable) the condition will always be true and the loop will never exit, therefore creating an infinite loop.
If you use the -Xmx flag appropriately when running java (see here for more info), you can increase your Heap size. This will allow you to use more memory if needed, in a controlled way. To my knowledge, there is not a way of "asking" for more heap memory from within a program itself (similarly to running the sbrk() or mmap() syscalls in C)
As the answer I linked to says:
For example, starting a JVM like so will start it with 256MB of memory, and will allow the process to use up to 2048MB of memory:
java -Xmx2048m -Xms256m
Also, you can use "k", "m", or "g" for Kilobytes, Megabytes and Gigabytes respectively. You cannot exceed 1GB (Heap size, that is), however, unless you are using the 64-bit JVM.
If you do the math with your particular use-case, assuming 64-bit longs * 100000000 costs about 800MB of space.
Does your program require you to store longs? If you were to use Integers instead of longs then you could store much more, either way, what requires you to store so many longs in an array?
Other solutions:
You could give your program more heap space when you start it with the argument -Xmx2G or some other length greater than the standard 512M or 1G
You could process a less number of array values, then save the array to hard drive. Then process the rest of the array and store it into the file.(Would require basic knowledge of Java's garbage collection)
You are trying to stuff a file with 468,000 lines into 5G of memory, and are running out of memory.
The data structure isn't the problem.
You need to change your approach and not do that. Process chunks of the file at a time, only extract the data you need, etc.
Inserting somewhere within an ArrayList won't give you amortized constant time, as the list will have to be copied internally - this will only work as long as you insert at the end.
Besides, when the ArrayList has to grow, it will calculate the new size by
int newCapacity = (oldCapacity * 3)/2 + 1;
which could waste huge amounts of memory in your case - it would be more efficient to use custom-sized String-arrays instead of the list (or call at least trimToSize() once you're done reading a column).
As long as you're only needing a few columns per time, I'd suggest to store each column in a separate file, which you can load/write on demand - if they'll only contain strings, you could think of some easy-readable binary format and use DataOutputStream and -InputStream, for instance. Inserting a column would simply become a file renaming operation... You could also add some caching, to keep the most recent or most often used columns in memory (Search for java.util.LinkedHashMap to get an idea of a simple LFU-Cache). Don't use a database if you don't need transactions or such, don't store such data with in a verbose format like XML - you'd get a huge performance loss otherwise.
Finally, I'd think about the content of the matrix, as strings can become pretty huge: Do you really need them as strings, or can you create a less memory consuming representation of them? For instance, if you'd only have 60.000 different strings, you could create a mapping between them and a short, and work with the shorts in memory.
This may sound glib, but your problem is that your application needs a List of 2.5 million rows as HashMaps.
This is an absurd, unreasonable and frankly ridiculous requirement; I can't imagine what use such a data structure would be good for.
Change the application to not require it.
You can try use byte[] instead of String object:
byte[] key = mappingKey.getBytes("UTF-8")
Each String object contains set of UTF-16 chars. It means 2-bytes per symbol in most case. UTF-8 encoding uses one byte for ASCII, two bytes for many europe languages.
Also each String object contains reference to char array. It means that you have two objects in memory heap: String and char array. Each object (even just new Object()) costs ~24 bytes (it depends from version Java VM and options).
So you can easy reduce count of objects by factor two (one byte[] instead of pair String + char[]), and array length of UTF-8 symbols usually less than length of UTF-16 chars.