I assume that you want the user to upload the file from some UI. Depending on the exact way in which you build UI, you might:
- Send a multipart HTTP POST request (mime type = multipart/form-data; see What should a Multipart HTTP request with multiple files look like?)
- Send a simple POST request with the body directly containing the file contents.
Either of the two can be fairly easily solved using Spring.
Assuming that we have the following entity:
@Data
@Entity
public class User {
@Id
private String username;
private String phoneNumber;
private String address;
}
And we define a Spring Data repository for accessing the database:
public interface UserRepository extends JpaRepository<User, String> {
}
For the CSV deserialization, I would propose using Jackson. Spring Boot already comes with Jackson, but we need to add a data format extension for CSV in your pom:
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-csv</artifactId>
</dependency>
This way, we can create a simple utility method that knows to read a CSV for a given POJO class:
public class CsvUtils {
private static final CsvMapper mapper = new CsvMapper();
public static <T> List<T> read(Class<T> clazz, InputStream stream) throws IOException {
CsvSchema schema = mapper.schemaFor(clazz).withHeader().withColumnReordering(true);
ObjectReader reader = mapper.readerFor(clazz).with(schema);
return reader.<T>readValues(stream).readAll();
}
}
And then we create a simple Rest Controller for handling the upload(s):
@RestController
@RequiredArgsConstructor
public class UserController {
private final UserRepository repository;
@PostMapping(value = "/upload", consumes = "text/csv")
public void uploadSimple(@RequestBody InputStream body) {
repository.saveAll(CsvUtils.read(User.class, body));
}
@PostMapping(value = "/upload", consumes = "multipart/form-data")
public void uploadMultipart(@RequestParam("file") MultipartFile file) {
repository.saveAll(CsvUtils.read(User.class, file.getInputStream()));
}
}
In case you also need some HTML for doing the upload, the following snippet is a minimal working example:
<form action="/upload" method="post" enctype="multipart/form-data">
<input type="file" name="file" id="file" />
<input type="submit" name="submit" value="Submit" />
</form>
Later edit: If you want to also validate the incoming data, first annotate your entity class attribute with javax.validation constraints. For example:
@Data
@Entity
class User {
@Id
@Email
@NotEmpty
private String username;
@Pattern(regexp = "[0-9 ()-]{4,12}")
private String phoneNumber;
private String address;
}
Then you can chose where do perform the actual validation call:
- Service level. This is what I personally recommend in this case, as it is fairly easy to setup and would perform the validations early enough. For this you introduce a simple
@Serviceclass between the controller and the repository.
@Service
@Validated
@RequiredArgsConstructor
class UserService {
private final UserRepository repository;
public void saveAll(@Valid List<User> users) {
repository.saveAll(users);
}
}
You would then use this service class instead of the repository inside the controller class.
Repository level: here you don't actually need to do anything. If you annotate your entity classes with validation constraints, Hibernate would automatically call the validation in a pre-insert listener (BeanValidationEventListener).
Controller level. This is trickier to setup. Move the CSV deserialization in a custom HttpMessageConverter. You should also add this converter to the FormHttpMessageConverter (such that it can use it to deserialize a part of the multi-part request). You could then theoretically just declare the
@Valid List<User>as inputs for your controller methods and Spring would automatically call the message converter based on the mime type and then call the validator. See Add JSON message converter for multipart/form-data for an example.
Lastly you can always manually call the validation whenever you want: Manually call Spring Annotation Validation.
Answer from Serban Petrescu on Stack OverflowSpring Boot Application Can't Find CSV File
java - Reading a csv file with Spring Boot application - Stack Overflow
How to read data from CSV file in Spring Boot or Java? - Stack Overflow
multithreading - Peformance issues reading CSV files in a Java (Spring Boot) application - Stack Overflow
Videos
repository for reference: https://github.com/izlemontee/pokemonserver
hey everyone, i'm building a pokemon database on Spring Boot but I can't even get it to start up.
What I intend to do is:
-in the repository, nationalDexRepo under the repo folder, read the Pokemon csv file (found in src/main/resources/static/csv/Pokemon.csv), and parse that info into a Map. I've done the parsing into a Map before when running as a normal java app, but when I try to run it as a Springboot app, it suddenly doesn't work. I included the reading of the CSV file in the constructor of the repo, nationalDexRepo.java to initialise it.
The repo contains these lines:
final String dirPath = "/csv"; final String csvName = "/Pokemon.csv";
and in the method, readCSV, the path is used:
public void readCSV()throws Exception{
boolean stop = false;
String fileString = dirPath+ csvName;
File f = new File(fileString);
System.out.println(fileString);
while(!stop){
if(f.exists() && f.isFile()){if the file does NOT exist, it's just a blank code block, and the fileString is "/csv/Pokemon.csv".
I tried different possible permutations of the directory, but still no dice. I'm still pretty new to SpringBoot so I'm not entirely sure if I'm calling the correct directories to access my file using the repo.
I don't think that splitting this work onto multiple threads is going to provide much improvement, and may in fact make the problem worse by consuming even more memory. The main problem is using too much heap memory, and the performance problem is likely to be due to excessive garbage collection when the remaining available heap is very small (but it's best to measure and profile to determine the exact cause of performance problems).
The memory consumption would be less from the replace and split operations, and more from the fact that the entire contents of the file need to be read into memory in this approach. Each line may not consume much memory, but multiplied by millions of lines, it all adds up.
If you have enough memory available on the machine to assign a heap size large enough to hold the entire contents, that will be the simplest solution, as it won't require changing the code.
Otherwise, the best way to deal with large amounts of data in a bounded amount of memory is to use a streaming approach. This means that each line of the file is processed and then passed directly to the output, without collecting all of the lines in memory in between. This will require changing the method signature to use a return type other than List. Assuming you are using Java 8 or later, the Stream API can be very helpful. You could rewrite the method like this:
public static Stream<List<String>> readCsv(InputStream inputStream) {
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
return reader.lines().map(line -> Arrays.asList(line.replace("\"", "").split(",")));
}
Note that this throws unchecked exceptions in case of an I/O error.
This will read and transform each line of input as needed by the caller of the method, and will allow previous lines to be garbage collected if they are no longer referenced. This then requires that the caller of this method also consume the data line by line, which can be tricky when generating JSON. The JakartaEE JsonGenerator API offers one possible approach. If you need help with this part of it, please open a new question including details of how you're currently generating JSON.
Instead of trying out a different approach, try to run with a profiler first and see where time is actually being spent. And use this information to change the approach.
Async-profiler is a very solid profiler (and free!) and will give you a very good impression of where time is being spent. And it will also show the time spend on garbage collection. So you can easily see the ratio of CPU utilization caused by garbage collection. It also has the ability to do allocation profiling to figure out which objects are being created (and where).
For a tutorial see the following link.
You can try this:
byte[] bytes = file.getBytes();
ByteArrayInputStream inputFilestream = new ByteArrayInputStream(bytes);
BufferedReader br = new BufferedReader(new InputStreamReader(inputFilestream ));
String line = "";
while ((line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
1.Model class = Listing.java
- Resource
.csvfile present underresources/static/listing.csv
private void createListingFromCsvRecord() {
String path = "classpath:static/listings.csv";
ResultSet rs = new Csv().read(path, null, null);
List<Listing> list = new ArrayList<>();
while (rs.next()) {
Listing listing = new Listing();
listing.setListingId(Long.parseLong(rs.getString("listingId")));
listing.setSiteArea(rs.getString("siteArea"));
list.add(listing);
}
myrepo.saveAll(list);
}