Would it take up far more space than simply writing a CSV file adapter that converts our List object to a CSV file?
It depends on the type of Row, and also on the size and other aspects of the data you are saving1.
On the one hand, the Java serialization protocol includes metadata for every class mentioned in the serialization. This takes significant space.
On the other hand:
- Java serialization only includes the metadata once per serialization. So if you serialize lots of instances of the same class, the metadata cost becomes insignificant.
- In a CSV file, all non textual data has to be converted to text. In some cases (e.g. large numbers, floating point numbers, booleans) the textual representation will be larger than the binary representation that is used in Java serialization.
1 - For example, an array of random numbers versus an array of zeros and ones. Java serialization will be better in the first case, and CSV will be better in the second.
But I think that you are probably focusing on the wrong thing here:
- Unless the files you are generating are enormous, the size probably doesn't matter. Disk space is cheap.
- The files are likely to be compressible in either case, with the less dense form probably being more compressible.
- What matters more is whether the representation is suitable for purpose; e.g.
- Do you want it to be human readable?
- Do you want it to be readable by non-Java programs, including shell scripts?
- Do you need to worry about changes to your Java code introducing class versus serialization version problems?
- Do you want to be able to stream the data? (When writing or reading.)
Would it take up far more space than simply writing a CSV file adapter that converts our List object to a CSV file?
It depends on the type of Row, and also on the size and other aspects of the data you are saving1.
On the one hand, the Java serialization protocol includes metadata for every class mentioned in the serialization. This takes significant space.
On the other hand:
- Java serialization only includes the metadata once per serialization. So if you serialize lots of instances of the same class, the metadata cost becomes insignificant.
- In a CSV file, all non textual data has to be converted to text. In some cases (e.g. large numbers, floating point numbers, booleans) the textual representation will be larger than the binary representation that is used in Java serialization.
1 - For example, an array of random numbers versus an array of zeros and ones. Java serialization will be better in the first case, and CSV will be better in the second.
But I think that you are probably focusing on the wrong thing here:
- Unless the files you are generating are enormous, the size probably doesn't matter. Disk space is cheap.
- The files are likely to be compressible in either case, with the less dense form probably being more compressible.
- What matters more is whether the representation is suitable for purpose; e.g.
- Do you want it to be human readable?
- Do you want it to be readable by non-Java programs, including shell scripts?
- Do you need to worry about changes to your Java code introducing class versus serialization version problems?
- Do you want to be able to stream the data? (When writing or reading.)
Java serialization will be less space efficient in certain cases than simply writing to a CSV file because it stores extra metadata to identify class-types.
I verified such a scenario with two simple test programs. The first one writes an array of ints to a .csv file.
import java.io.*;
public class CSVDemo {
public static void main(String [] args) {
try {
PrintWriter pw = new PrintWriter(new File("dummy.csv"));
StringBuilder sb = new StringBuilder();
for(int i = 0; i < 1000; i++){
sb.append(1);
sb.append(",");
}
pw.write(sb.toString());
pw.close();
System.out.printf("Data is saved in dummy.csv");
} catch(FileNotFoundException e) {
e.printStackTrace();
}
}
}
The second one serializes an object containing an array of ints to a .ser file.
import java.io.*;
public class SerializeDemo
{
public static void main(String [] args)
{
DummyData dummy = new DummyData();
try {
FileOutputStream fileOut = new FileOutputStream("dummy.ser");
ObjectOutputStream out = new ObjectOutputStream(fileOut);
out.writeObject(dummy);
out.close();
fileOut.close();
System.out.printf("Serialized data is saved in dummy.ser");
} catch(IOException i) {
i.printStackTrace();
}
}
public static class DummyData implements java.io.Serializable{
int[] data = new int[1000];
public DummyData(){
for(int i = 0; i < 1000; i++){
data[i] = 1;
}
}
}
}
The .ser file took 4079 bytes. The .csv file took 2000 bytes. Granted, this is a slight simplification of your use case (I'm equating an int to your Row type), but the general trend should be the same.
Trying with larger numbers yields the same result. Using 100000 ints results in ~400KB for .ser and 200KB for .csv
However, as the below comment pointed out, if choosing random values for ints, the .csv actually grows larger.
You might also want to have a look at FST.
also provides tools for offheap reading/writing
have a look at kryo.
its much much faster than the built-in serialization mechanism (that writes out a lot of strings and relies heavily on reflection), but a bit harder to use.
edit: R.Moeller below suggested FST, which i've never heard of until now but looks to be both faster than kryo and compatible with java built-in serialization (which should make it even easier to use), so i'd look at that 1st