1: Now I want to ask why writer use int c? even we are reading characters.
FileInputStream.read() returns one byte of data as an int. This works because a byte can be represented as an int without loss of precision. See this answer to understand why int is returned instead of byte.
2: The second why use -1 in while condition?
When the end of file is reached, -1 is returned.
3: How out.write(c); method convert int to again characters? that provide same output in outagain.txt file
FileOutputStream.write() takes a byte parameter as an int. Since an int spans over more values than a byte, the 24 high-order bits of the given int are ignored, making it a byte-compatible value: an int in Java is always 32 bits. By removing the 24 high-order bits, you're down to a 8 bits value, i.e. a byte.
I suggest you read carefully the Javadocs for each of those method. As reference, they answer all of your questions:
read:
Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
write:
Answer from Tunaki on Stack OverflowWrites the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.
file - How FileInputStream and FileOutputStream Works in Java? - Stack Overflow
7. What are FileInputStream and FileOutputStream classes?
Java FileInputStream FileOutputStream difference in the run - Stack Overflow
What is difference between FileOutputStream and ObjectOutputStream?
Videos
1: Now I want to ask why writer use int c? even we are reading characters.
FileInputStream.read() returns one byte of data as an int. This works because a byte can be represented as an int without loss of precision. See this answer to understand why int is returned instead of byte.
2: The second why use -1 in while condition?
When the end of file is reached, -1 is returned.
3: How out.write(c); method convert int to again characters? that provide same output in outagain.txt file
FileOutputStream.write() takes a byte parameter as an int. Since an int spans over more values than a byte, the 24 high-order bits of the given int are ignored, making it a byte-compatible value: an int in Java is always 32 bits. By removing the 24 high-order bits, you're down to a 8 bits value, i.e. a byte.
I suggest you read carefully the Javadocs for each of those method. As reference, they answer all of your questions:
read:
Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
write:
Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.
Just read the docs.
here is the read method docs http://docs.oracle.com/javase/7/docs/api/java/io/FileInputStream.html#read()
public int read() throws IOException Reads a byte of data from this input stream. This method blocks if no input is yet available.
Specified by: read in class InputStream
Returns: the next byte of data, or -1 if the end of the file is reached.
That int is a your next set of bytes data. Now , here are the answers.
1) When you assign a char to an int, it denotes it's ascii number to the int.
If you are interested, here us the list of chars and their ascii codes https://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/ascii.html
2)-1 if the end of the file is reached. So that's a check to data exists or not.
3)When you send an ascii code to print writer, it's prints that corresponding char to the file.
FileInputStream 's read() method follows this logic:
Reads a byte of data from this input stream. This method blocks if no input is yet available.
So assigning the value of its return to a variable, such as:
while((len = fis.read())!= -1)
Is avoiding the byte of data just read from the stream to be forgotten, as every read() call will be assigned to your len variable.
Instead, this code bypasses one of every two bytes from the stream, as the read() executed in the while condition is never assigned to a variable. So the stream advances without half of the bytes being read (assigned to len):
while (fis.read() != -1) { // reads a byte of data (but not saved)
int len = fis.read(); // next byte of data saved
fos.write(len); // possible -1 written here
}
@aran and others already pointed out the solution to your problem.
However there are more sides to this, so I extended your example:
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
public class FileCopyFisFos {
public static void main(final String[] args) throws IOException {
final File src = new File("d:/Test1/OrigFile.MP4");
final File sink = new File("d:/Test2/DestFile.mp4");
{
final long startMS = System.currentTimeMillis();
final long bytesCopied = copyFileSimple(src, sink);
System.out.println("Simple copy transferred " + bytesCopied + " bytes in " + (System.currentTimeMillis() - startMS) + "ms");
}
{
final long startMS = System.currentTimeMillis();
final long bytesCopied = copyFileSimpleFaster(src, sink);
System.out.println("Simple+Fast copy transferred " + bytesCopied + " bytes in " + (System.currentTimeMillis() - startMS) + "ms");
}
{
final long startMS = System.currentTimeMillis();
final long bytesCopied = copyFileFast(src, sink);
System.out.println("Fast copy transferred " + bytesCopied + " bytes in " + (System.currentTimeMillis() - startMS) + "ms");
}
System.out.println("Test completed.");
}
static public long copyFileSimple(final File pSourceFile, final File pSinkFile) throws IOException {
try (
final FileInputStream fis = new FileInputStream(pSourceFile);
final FileOutputStream fos = new FileOutputStream(pSinkFile);) {
long totalBytesTransferred = 0;
while (true) {
final int readByte = fis.read();
if (readByte < 0) break;
fos.write(readByte);
++totalBytesTransferred;
}
return totalBytesTransferred;
}
}
static public long copyFileSimpleFaster(final File pSourceFile, final File pSinkFile) throws IOException {
try (
final FileInputStream fis = new FileInputStream(pSourceFile);
final FileOutputStream fos = new FileOutputStream(pSinkFile);
BufferedInputStream bis = new BufferedInputStream(fis);
BufferedOutputStream bos = new BufferedOutputStream(fos);) {
long totalBytesTransferred = 0;
while (true) {
final int readByte = bis.read();
if (readByte < 0) break;
bos.write(readByte);
++totalBytesTransferred;
}
return totalBytesTransferred;
}
}
static public long copyFileFast(final File pSourceFile, final File pSinkFile) throws IOException {
try (
final FileInputStream fis = new FileInputStream(pSourceFile);
final FileOutputStream fos = new FileOutputStream(pSinkFile);) {
long totalBytesTransferred = 0;
final byte[] buffer = new byte[20 * 1024];
while (true) {
final int bytesRead = fis.read(buffer);
if (bytesRead < 0) break;
fos.write(buffer, 0, bytesRead);
totalBytesTransferred += bytesRead;
}
return totalBytesTransferred;
}
}
}
The hints that come along with that code:
- There is the java.nio package that usualy does those things a lot faster and in less code.
- Copying single bytes is 1'000-40'000 times slower that bulk copy.
- Using try/resource/catch is the best way to avoid problems with reserved/locked resources like files etc.
- If you solve something that is quite commonplace, I suggest you put it in a utility class of your own or even your own library.
- There are helper classes like BufferedInputStream and BufferedOutputStream that take care of efficiency greatly; see example copyFileSimpleFaster().
- But as usual, it is the quality of the concept that has the most impact on the implementation; see example copyFileFast().
- There are even more advanced concepts (similar to java.nio), that take into account concepts like OS caching behaviour etc, which will give performance another kick.
Check my outputs, or run it on your own, to see the differences in performance:
Simple copy transferred 1608799 bytes in 12709ms
Simple+Fast copy transferred 1608799 bytes in 51ms
Fast copy transferred 1608799 bytes in 4ms
Test completed.
As in the title, i can not understand the difference between two these class. For example, why we combining them like in the sample code below?
import java.io.Serializable;
public class Model implements Serializable {
...
}
Model m = ....;
File outFile = new File(saveFileName);
try {ObjectOutputStream out = new ObjectOutputStream(newFileOutputStream(outFile));
out.writeObject(m);
out.close();
} catch (IOException excp) { ... }The classes FSDataInputStream and FSDataOutputStream are classes in hadoop-common
FSDataInputStream
FSDataInputStream adds high performance APIs for reading data at specific offsets into byte arrays (PositionedReadable) or bytebuffers. these are used extensively in libraries reading files where the reads are not sequential, more random IO. parquet, orc etc. FileSystem implementations often provide highly efficient implementations of these. These apis are not relevant for simple file copy unless you really are trying for maximum performance, have opened the same source file to multiple streams and are fetching blocks in parallel across them. Distcp does things like this, which is why it can overload networks if you try hard.
full specification, including of PositionedReadable: fsdatainputstream
FSDataOutputStream
FSDataOutputStream doesn't add that much to the normal DataOutputStream; the most important interface is Syncable, whose hflush and hsync calls have specific guarantees about durability, as in "when they return, the data has been persisted to HDFS or the other filesystem, for hsync, all the way to disk". If you are implementing a database like HBase, you need these and those guarantees. If you aren't, you really don't. In recent hadoop releases, trying to use them when writing to S3 will simply log a warning message telling you to stop it. It's not a real file system, after all.
full specification, including of syncable: outputstream
copying files in spark at scale
IF you want to copy files efficiently in spark, open the source files and dest files with a buffer of a few MB, read into the buffer, then write it back. you can distribute this work across the cluster for better parallelism, as this example does: https://github.com/hortonworks-spark/cloud-integration/blob/master/spark-cloud-integration/src/main/scala/com/cloudera/spark/cloud/applications/CloudCp.scala
- If you just want to copy one or two files, just do it in a single process, maybe multithreaded.
- If you are really seeking performance against s3, take the list of files to copy, schedule the largest few files first so they don't hold you at the end, then randomize the rest of the list to avoid creating hotspots in the s3 bucket.
You can use either, it just depends on what you need.
They both are just stream implementations. Ultimately what you will be doing is taking and inputstream from one bucket and writing it to the outputstream of another.
The FileInputStream and FileOutputStream are concrete components that provide the functionality to read and write streams from a mapped file.
The FSDataInputStream and FSDataOutputStream are concrete decorators of an inputstream. Meaning the provide or decorate the inputstream with functionality, such as reading and writing primitives and providing buffered streams.
Which one to choose? Do you need a the decorations provided by FSDataOutputStream and FSDataInputStream? Is FileInputStream and FileOutputStream sufficient?
Personally, I would look to use Readers and Writers as demonstrated here:
How can I read an AWS S3 File with Java?
private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();
private Collection<String> loadFileFromS3() {
try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME,
FILE_NAME);
final InputStreamReader streamReader = new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
final BufferedReader reader = new BufferedReader(streamReader)) {
return reader.lines().collect(Collectors.toSet());
} catch (final IOException e) {
log.error(e.getMessage(), e)
return Collections.emptySet();
}
}
The statement FileWriter fos = new FileWriter(testFile); truncates the existing file.
It does not make sense for you to use streaming access to read and write the same file, as this won't give reliable results. Use RandomAccessFile if you want to read / write the same file: this has calls to seek current position and perform read or writes at different positions of a file.
https://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html
FileWriter actually deletes everything in a file before writing. To preserve the text, use
new FileWriter(file, true);
The true parameter is the append parameter of the filewriter. Otherwise it will just overwrite everything