"How can I directly (without saving the file on 2nd server) download the file from 1st server to client's machine?"
Just use the Client API and get the InputStream from the response
Client client = ClientBuilder.newClient();
String url = "...";
final InputStream responseStream = client.target(url).request().get(InputStream.class);
There are two flavors to get the InputStream. You can also use
Response response = client.target(url).request().get();
InputStream is = (InputStream)response.getEntity();
Which one is the more efficient? I'm not sure, but the returned InputStreams are different classes, so you may want to look into that if you care to.
From 2nd server I can get a ByteArrayOutputStream to get the file from 1st server, can I pass this stream further to the client using the REST service?
So most of the answers you'll see in the link provided by @GradyGCooper seem to favor the use of StreamingOutput. An example implementation might be something like
final InputStream responseStream = client.target(url).request().get(InputStream.class);
System.out.println(responseStream.getClass());
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) throws IOException, WebApplicationException {
int length;
byte[] buffer = new byte[1024];
while((length = responseStream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
out.flush();
responseStream.close();
}
};
return Response.ok(output).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
But if we look at the source code for StreamingOutputProvider, you'll see in the writeTo, that it simply writes the data from one stream to another. So with our implementation above, we have to write twice.
How can we get only one write? Simple return the InputStream as the Response
final InputStream responseStream = client.target(url).request().get(InputStream.class);
return Response.ok(responseStream).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
If we look at the source code for InputStreamProvider, it simply delegates to ReadWriter.writeTo(in, out), which simply does what we did above in the StreamingOutput implementation
public static void writeTo(InputStream in, OutputStream out) throws IOException {
int read;
final byte[] data = new byte[BUFFER_SIZE];
while ((read = in.read(data)) != -1) {
out.write(data, 0, read);
}
}
Asides:
Clientobjects are expensive resources. You may want to reuse the sameClientfor request. You can extract aWebTargetfrom the client for each request.WebTarget target = client.target(url); InputStream is = target.request().get(InputStream.class);I think the
WebTargetcan even be shared. I can't find anything in the Jersey 2.x documentation (only because it is a larger document, and I'm too lazy to scan through it right now :-), but in the Jersey 1.x documentation, it says theClientandWebResource(which is equivalent toWebTargetin 2.x) can be shared between threads. So I'm guessing Jersey 2.x would be the same. but you may want to confirm for yourself.You don't have to make use of the
ClientAPI. A download can be easily achieved with thejava.netpackage APIs. But since you're already using Jersey, it doesn't hurt to use its APIsThe above is assuming Jersey 2.x. For Jersey 1.x, a simple Google search should get you a bunch of hits for working with the API (or the documentation I linked to above)
UPDATE
I'm such a dufus. While the OP and I are contemplating ways to turn a ByteArrayOutputStream to an InputStream, I missed the simplest solution, which is simply to write a MessageBodyWriter for the ByteArrayOutputStream
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
@Provider
public class OutputStreamWriter implements MessageBodyWriter<ByteArrayOutputStream> {
@Override
public boolean isWriteable(Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return ByteArrayOutputStream.class == type;
}
@Override
public long getSize(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return -1;
}
@Override
public void writeTo(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType,
MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream)
throws IOException, WebApplicationException {
t.writeTo(entityStream);
}
}
Then we can simply return the ByteArrayOutputStream in the response
return Response.ok(baos).build();
D'OH!
UPDATE 2
Here are the tests I used (
Resource class
@Path("test")
public class TestResource {
final String path = "some_150_mb_file";
@GET
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response doTest() throws Exception {
InputStream is = new FileInputStream(path);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int len;
byte[] buffer = new byte[4096];
while ((len = is.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, len);
}
System.out.println("Server size: " + baos.size());
return Response.ok(baos).build();
}
}
Client test
public class Main {
public static void main(String[] args) throws Exception {
Client client = ClientBuilder.newClient();
String url = "http://localhost:8080/api/test";
Response response = client.target(url).request().get();
String location = "some_location";
FileOutputStream out = new FileOutputStream(location);
InputStream is = (InputStream)response.getEntity();
int len = 0;
byte[] buffer = new byte[4096];
while((len = is.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.flush();
out.close();
is.close();
}
}
UPDATE 3
So the final solution for this particular use case was for the OP to simply pass the OutputStream from the StreamingOutput's write method. Seems the third-party API, required a OutputStream as an argument.
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) {
thirdPartyApi.downloadFile(.., .., .., out);
}
}
return Response.ok(output).build();
Not quite sure, but seems the reading/writing within the resource method, using ByteArrayOutputStream`, realized something into memory.
The point of the downloadFile method accepting an OutputStream is so that it can write the result directly to the OutputStream provided. For instance a FileOutputStream, if you wrote it to file, while the download is coming in, it would get directly streamed to the file.
It's not meant for us to keep a reference to the OutputStream, as you were trying to do with the baos, which is where the memory realization comes in.
So with the way that works, we are writing directly to the response stream provided for us. The method write doesn't actually get called until the writeTo method (in the MessageBodyWriter), where the OutputStream is passed to it.
You can get a better picture looking at the MessageBodyWriter I wrote. Basically in the writeTo method, replace the ByteArrayOutputStream with StreamingOutput, then inside the method, call streamingOutput.write(entityStream). You can see the link I provided in the earlier part of the answer, where I link to the StreamingOutputProvider. This is exactly what happens
"How can I directly (without saving the file on 2nd server) download the file from 1st server to client's machine?"
Just use the Client API and get the InputStream from the response
Client client = ClientBuilder.newClient();
String url = "...";
final InputStream responseStream = client.target(url).request().get(InputStream.class);
There are two flavors to get the InputStream. You can also use
Response response = client.target(url).request().get();
InputStream is = (InputStream)response.getEntity();
Which one is the more efficient? I'm not sure, but the returned InputStreams are different classes, so you may want to look into that if you care to.
From 2nd server I can get a ByteArrayOutputStream to get the file from 1st server, can I pass this stream further to the client using the REST service?
So most of the answers you'll see in the link provided by @GradyGCooper seem to favor the use of StreamingOutput. An example implementation might be something like
final InputStream responseStream = client.target(url).request().get(InputStream.class);
System.out.println(responseStream.getClass());
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) throws IOException, WebApplicationException {
int length;
byte[] buffer = new byte[1024];
while((length = responseStream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
out.flush();
responseStream.close();
}
};
return Response.ok(output).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
But if we look at the source code for StreamingOutputProvider, you'll see in the writeTo, that it simply writes the data from one stream to another. So with our implementation above, we have to write twice.
How can we get only one write? Simple return the InputStream as the Response
final InputStream responseStream = client.target(url).request().get(InputStream.class);
return Response.ok(responseStream).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
If we look at the source code for InputStreamProvider, it simply delegates to ReadWriter.writeTo(in, out), which simply does what we did above in the StreamingOutput implementation
public static void writeTo(InputStream in, OutputStream out) throws IOException {
int read;
final byte[] data = new byte[BUFFER_SIZE];
while ((read = in.read(data)) != -1) {
out.write(data, 0, read);
}
}
Asides:
Clientobjects are expensive resources. You may want to reuse the sameClientfor request. You can extract aWebTargetfrom the client for each request.WebTarget target = client.target(url); InputStream is = target.request().get(InputStream.class);I think the
WebTargetcan even be shared. I can't find anything in the Jersey 2.x documentation (only because it is a larger document, and I'm too lazy to scan through it right now :-), but in the Jersey 1.x documentation, it says theClientandWebResource(which is equivalent toWebTargetin 2.x) can be shared between threads. So I'm guessing Jersey 2.x would be the same. but you may want to confirm for yourself.You don't have to make use of the
ClientAPI. A download can be easily achieved with thejava.netpackage APIs. But since you're already using Jersey, it doesn't hurt to use its APIsThe above is assuming Jersey 2.x. For Jersey 1.x, a simple Google search should get you a bunch of hits for working with the API (or the documentation I linked to above)
UPDATE
I'm such a dufus. While the OP and I are contemplating ways to turn a ByteArrayOutputStream to an InputStream, I missed the simplest solution, which is simply to write a MessageBodyWriter for the ByteArrayOutputStream
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
@Provider
public class OutputStreamWriter implements MessageBodyWriter<ByteArrayOutputStream> {
@Override
public boolean isWriteable(Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return ByteArrayOutputStream.class == type;
}
@Override
public long getSize(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return -1;
}
@Override
public void writeTo(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType,
MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream)
throws IOException, WebApplicationException {
t.writeTo(entityStream);
}
}
Then we can simply return the ByteArrayOutputStream in the response
return Response.ok(baos).build();
D'OH!
UPDATE 2
Here are the tests I used (
Resource class
@Path("test")
public class TestResource {
final String path = "some_150_mb_file";
@GET
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response doTest() throws Exception {
InputStream is = new FileInputStream(path);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int len;
byte[] buffer = new byte[4096];
while ((len = is.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, len);
}
System.out.println("Server size: " + baos.size());
return Response.ok(baos).build();
}
}
Client test
public class Main {
public static void main(String[] args) throws Exception {
Client client = ClientBuilder.newClient();
String url = "http://localhost:8080/api/test";
Response response = client.target(url).request().get();
String location = "some_location";
FileOutputStream out = new FileOutputStream(location);
InputStream is = (InputStream)response.getEntity();
int len = 0;
byte[] buffer = new byte[4096];
while((len = is.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.flush();
out.close();
is.close();
}
}
UPDATE 3
So the final solution for this particular use case was for the OP to simply pass the OutputStream from the StreamingOutput's write method. Seems the third-party API, required a OutputStream as an argument.
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) {
thirdPartyApi.downloadFile(.., .., .., out);
}
}
return Response.ok(output).build();
Not quite sure, but seems the reading/writing within the resource method, using ByteArrayOutputStream`, realized something into memory.
The point of the downloadFile method accepting an OutputStream is so that it can write the result directly to the OutputStream provided. For instance a FileOutputStream, if you wrote it to file, while the download is coming in, it would get directly streamed to the file.
It's not meant for us to keep a reference to the OutputStream, as you were trying to do with the baos, which is where the memory realization comes in.
So with the way that works, we are writing directly to the response stream provided for us. The method write doesn't actually get called until the writeTo method (in the MessageBodyWriter), where the OutputStream is passed to it.
You can get a better picture looking at the MessageBodyWriter I wrote. Basically in the writeTo method, replace the ByteArrayOutputStream with StreamingOutput, then inside the method, call streamingOutput.write(entityStream). You can see the link I provided in the earlier part of the answer, where I link to the StreamingOutputProvider. This is exactly what happens
See example here: Input and Output binary streams using JERSEY?
Pseudo code would be something like this (there are a few other similar options in above mentioned post):
@Path("file/")
@GET
@Produces({"application/pdf"})
public StreamingOutput getFileContent() throws Exception {
public void write(OutputStream output) throws IOException, WebApplicationException {
try {
//
// 1. Get Stream to file from first server
//
while(<read stream from first server>) {
output.write(<bytes read from first server>)
}
} catch (Exception e) {
throw new WebApplicationException(e);
} finally {
// close input stream
}
}
}
Option 1 using an InputStreamResource
Resource implementation for a given InputStream.
Should only be used if no other specific Resource implementation is > applicable. In particular, prefer ByteArrayResource or any of the file-based Resource implementations where possible.
@RequestMapping(path = "/download", method = RequestMethod.GET)
public ResponseEntity<Resource> download(String param) throws IOException {
// ...
InputStreamResource resource = new InputStreamResource(new FileInputStream(file));
return ResponseEntity.ok()
.headers(headers)
.contentLength(file.length())
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(resource);
}
Option2 as the documentation of the InputStreamResource suggests - using a ByteArrayResource:
@RequestMapping(path = "/download", method = RequestMethod.GET)
public ResponseEntity<Resource> download(String param) throws IOException {
// ...
Path path = Paths.get(file.getAbsolutePath());
ByteArrayResource resource = new ByteArrayResource(Files.readAllBytes(path));
return ResponseEntity.ok()
.headers(headers)
.contentLength(file.length())
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(resource);
}
The below Sample code worked for me and might help someone.
import org.springframework.core.io.ByteArrayResource;
import org.springframework.core.io.Resource;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
@RestController
@RequestMapping("/app")
public class ImageResource {
private static final String EXTENSION = ".jpg";
private static final String SERVER_LOCATION = "/server/images";
@RequestMapping(path = "/download", method = RequestMethod.GET)
public ResponseEntity<Resource> download(@RequestParam("image") String image) throws IOException {
File file = new File(SERVER_LOCATION + File.separator + image + EXTENSION);
HttpHeaders header = new HttpHeaders();
header.add(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=img.jpg");
header.add("Cache-Control", "no-cache, no-store, must-revalidate");
header.add("Pragma", "no-cache");
header.add("Expires", "0");
Path path = Paths.get(file.getAbsolutePath());
ByteArrayResource resource = new ByteArrayResource(Files.readAllBytes(path));
return ResponseEntity.ok()
.headers(header)
.contentLength(file.length())
.contentType(MediaType.parseMediaType("application/octet-stream"))
.body(resource);
}
}
As you noticed yourself, you're dealing with binary data here. So you shouldn't create a String from your response. Better get the input stream and pipe it to your file.
Response response = invokeDownload(authToken, url);
InputStream in = response.readEntity(InputStream.class);
Path path = Paths.get(savePath, filename);
Files.copy(in, path);
1) I assume by this point you're clear on the difference between "binary file" and "text file". And that you can only capture the latter into a "string".
2) Sebastian gave you excellent advice for capturing a binary file (+1, Sebastian!). VERY IMPORTANT: you should always set the MIME type (Content-Type: xxx/yyy)in cases like this. Here is another link that might be useful.
3) Finally, there are cases where you might WANT to treat "binary" data as text. This is how e-mail attachments work with SMTP (a text protocol). In these cases, you want to use Base64 Encoding. For example: JAX-RS | Download PDF from Base64 encoded data