Looking at the implementation of Buffer.concat: It seems to be doing exactly what you suggested as replacement in implementation 2/first case: Create a new buffer with the calculated total length, then copy all buffers into the output.
So, based on that it seems reasonable to continue to use Buffer.concat directly, because the code is more concise, easier to follow, and describes better what you want to do ("concatenate all these buffers!")
For the writer case: Your suggested implementations aren't equivalent -- the first one would leave space at the end for adding new data, while the second implementation provides exactly the space needed for the existing buffers. If you need the extra space (optimization for expected future writes, for example), then stick to the first variant, otherwise the same argument applies: Using concat is more concise and easier to read.
node.js - Buffer.concat vs Buffer.copy in nodejs - Stack Overflow
node.js Buffers concat is driving me nuts
How to concat buffers with delimiter in node.js?
Buffer.concatenate only produces buffer of last buffer in the array
» npm install buffer-concat
Hello reddit,
i am currently investigating in a bug that is driving me crazy the whole day. Googling, Stackoverflow, every blog says that the Buffer.concat(list, length) function is there to achieve the appending of all buffers to a single one. But all i get after concatenating multiple buffers is the first one.
I am fetching small .wav files from a server and try to append them to a single new buffer and send it over to the frontend. Those files are small 2 seconds samples.
On the frontend i want to turn that buffer into an AudioBuffer. Which works, but only plays the first buffer of the buffer list.
I use node-fetch for that on the express.js backend
fetch(url).then(response => response.buffer());
I do that for a number of 1-20 files and store those buffers in an array.
I checked every buffer in that array and wrote it to a file on the server. Every single file is playable and unique.
But if i do that
let concatedBuffers = Buffer.concat(allBuffers, totalLength);
and write that big buffer ( got the size of all buffers combined, still below 5 MB) to a file, it only represents the first buffer item.
I tried to use a library for buffer concat, i even wrote an implementation. But still the same result. Only the first buffer is there after appending.
Does someone have a clue and can point out my misleading understanding of Buffers, please?
The implementation ( copied from buffer-concat library):function concatenateBuffers( list, length? ){let size = 0;if (!Array.isArray(list)){throw new Error('Usage: concatenateBuffers(list, [length])');}
if (list.length === 0){return new Buffer(0);} else if (list.length === 1){return list[0];}
if (typeof length !== 'number'){size = 0;for (let i = 0; i < list.length; i++){let buf = list[i];size += buf.length;}
console.log('calculated Size: ', size);} else {size = length;}
let buffer = new Buffer(size);let pos = 0;for (let i = 0; i < list.length; i++){let buf = list[i];buf.copy(buffer, pos);pos += buf.length;}return buffer;}
There are three changes I would suggest.
First, if you are using Node v6, use Buffer.from() instead of new Buffer(), as the latter is deprecated.
Second, you don't need to pass an argument for totalLength to Buffer.concat(), since it will be calculated automatically from the length of all of the buffers passed. While the docs note it will be faster to pass a total length, this will really only be true if you pass a constant value. What you are doing above is computing the length and then passing that, which is what the concat() function will do internally anyway.
Finally, I would recommend putting this in a function that works like Array.prototype.join(), but for buffers.
function joinBuffers(buffers, delimiter = ' ') {
let d = Buffer.from(delimiter);
return buffers.reduce((prev, b) => Buffer.concat([prev, d, b]));
}
And you can use it like this:
let buf1 = Buffer.from('Foo');
let buf2 = Buffer.from('Bar');
let buf3 = Buffer.from('Baz');
let joined = joinBuffers([buf1, buf2, buf3]);
console.log(joined.toString()); // Foo Bar Baz
Or set a custom delimiter like this:
let joined2 = joinBuffers([buf1, buf2, buf3], ' and ');
console.log(joined2.toString()); // Foo and Bar and Baz
Read the Buffer stream and save it to file as:
const data = [];
req.on('data', stream => {
data.push(stream);
});
req.on('close', () => {
const parsedData = Buffer.concat(data).toString('utf8');
fs.writeFileSync('./test.text', parsedData);
});
Rather than concatenating every time (which creates a new buffer each time), just keep an array of all of your buffers and concat at the end.
Buffer.concat() can take a whole list of buffers. Then it's done in one operation. https://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength
If you read from a file and know the size of that file, then you can pre-allocate the final buffer. Then each time you get a chunk of data, you can simply write it to that large 16Mb buffer.
// use the "unsafe" version to avoid clearing 16Mb for nothing
let buf = Buffer.allocUnsafe(file_size)
let pos = 0
file.on('data', (chunk) => {
buf.fill(chunk, pos, pos + chunk.length)
pos += chunk.length
})
if(pos != file_size) throw new Error('Ooops! something went wrong.')
The main difference with @Brad's code sample is that you're going to use 16Mb + size of one chunk (roughly) instead of 32Mb + size of one chunk.
Also, each chunk has a header, various pointers, etc. so you are not unlikely to use 33Mb or even 34Mb... that's a lot more RAM. The amount of RAM copied is otherwise the same. That being said, it could be that Node starts reading the next chunk while you copy so it could make it transparent. When done in one large chunk in the 'end' event, you're going to have to wait for the contact() to complete while doing nothing else in parallel.
In case you are receiving an HTTP POST and are reading it. Remember that you get a Content-Length parameter so you also have the length in that case and can pre-allocate the entire buffer before reading the data.