Give Java NIO a try:
URL website = new URL("http://www.website.com/information.asp");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("information.html");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
Using transferFrom() is potentially much more efficient than a simple loop that reads from the source channel and writes to this channel. Many operating systems can transfer bytes directly from the source channel into the filesystem cache without actually copying them.
Check more about it here.
Note: The third parameter in transferFrom is the maximum number of bytes to transfer. Integer.MAX_VALUE will transfer at most 2^31 bytes, Long.MAX_VALUE will allow at most 2^63 bytes (larger than any file in existence).
Give Java NIO a try:
URL website = new URL("http://www.website.com/information.asp");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("information.html");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
Using transferFrom() is potentially much more efficient than a simple loop that reads from the source channel and writes to this channel. Many operating systems can transfer bytes directly from the source channel into the filesystem cache without actually copying them.
Check more about it here.
Note: The third parameter in transferFrom is the maximum number of bytes to transfer. Integer.MAX_VALUE will transfer at most 2^31 bytes, Long.MAX_VALUE will allow at most 2^63 bytes (larger than any file in existence).
Use Apache Commons IO. It is just one line of code:
FileUtils.copyURLToFile(URL, File)
Is there a Java file download workflow that works with both web URLs and file URLs (file://)?
You could pass an InputStream instead of an url into your download method and pass that one into Channels.newChannel().
More on reddit.comHow to download a file that gets processed over an HTTP request in Java?
Download file from google drive with URL
How to fetch data and download file from website in java?
Videos
The term project I came up with for a Java course revolves around being able to download files from the internet, but apparently AutoBot, the automated testing system that I have to pass, doesn't let you connect to the internet. The only solution I can think of is to use file URLs (for example, file://data/test/file.xml) in the test cases to bypass the network connection all together. However, the built-in method I'm using for downloads doesn't support file urls.
Here is one of my download functions:
public void download(File path, URL url) throws IOException {
prepareDirectory(new File(path.getParent()));
ReadableByteChannel channel = Channels.newChannel(url.openStream());
FileOutputStream stream = new FileOutputStream(path);
stream.getChannel().transferFrom(channel, 0, Long.MAX_VALUE);
}But it throws "java.net.UnknownHostException: data" in the url.openStream() method call. I think the problem is that I need to generate an InputStream from a file URL.
Is there a downloading workflow that works with both http and file URLs? For a file url, I just want it to copy the file contents into the download destination, or anything really, as long as it accepts both kinds of URLs in one constructor or method call so I get 100% line/branch test coverage, and it writes the right data to the target path. I'm allowed to use third party open source libraries so something not in the standard library is also okay.
Failing that, how should I test something like this without an internet connection? My TAs don't even know for sure how to do this correctly.
It's ironic that my code actually works correctly, except when testing it due to an arbitrary limitation.