Update 2016 - five years on there are now new methods in the specs (see support below) to convert between strings and typed arrays using proper encoding.
##TextEncoder
The TextEncoder represents:
The
TextEncoderinterface represents an encoder for a specific method, that is a specific character encoding, liarraybufferkeutf-8,An encoder takes a stream of code points as input and emits a stream of bytes.iso-8859-2,koi8,cp1261,gbk, ...
Change note since the above was written: (ibid.)
Note: Firefox, Chrome and Opera used to have support for encoding types other than utf-8 (such as utf-16, iso-8859-2, koi8, cp1261, and gbk). As of Firefox 48 [...], Chrome 54 [...] and Opera 41, no other encoding types are available other than utf-8, in order to match the spec.*
*) Updated specs (W3) and here (whatwg).
After creating an instance of the TextEncoder it will take a string and encode it using a given encoding parameter:
if (!("TextEncoder" in window))
alert("Sorry, this browser does not support TextEncoder...");
var enc = new TextEncoder(); // always utf-8
console.log(enc.encode("This is a a string to be converted to a Uint8Array"));
You then of course use the .buffer parameter on the resulting Uint8Array to convert the underlaying ArrayBuffer to a different view if needed.
Just make sure that the characters in the string adhere to the encoding schema, for example, if you use characters outside the UTF-8 range in the example they will be encoded to two bytes instead of one.
For general use you would use UTF-16 encoding for things like localStorage.
##TextDecoder
Likewise, the opposite process uses the TextDecoder:
The
TextDecoderinterface represents a decoder for a specific method, that is a specific character encoding, likeutf-8,iso-8859-2,koi8,cp1261,gbk, ... A decoder takes a stream of bytes as input and emits a stream of code points.
All available decoding types can be found here.
if (!("TextDecoder" in window))
alert("Sorry, this browser does not support TextDecoder...");
var enc = new TextDecoder("utf-8");
var arr = new Uint8Array([84,104,105,115,32,105,115,32,97,32,85,105,110,116,
56,65,114,114,97,121,32,99,111,110,118,101,114,116,
101,100,32,116,111,32,97,32,115,116,114,105,110,103]);
console.log(enc.decode(arr));
##The MDN StringView library
An alternative to these is to use the StringView library (licensed as lgpl-3.0) which goal is:
- to create a C-like interface for strings (i.e., an array of character codes — an ArrayBufferView in JavaScript) based upon the JavaScript ArrayBuffer interface
- to create a highly extensible library that anyone can extend by adding methods to the object StringView.prototype
- to create a collection of methods for such string-like objects (since now: stringViews) which work strictly on arrays of numbers rather than on creating new immutable JavaScript strings
- to work with Unicode encodings other than JavaScript's default UTF-16 DOMStrings
giving much more flexibility. However, it would require us to link to or embed this library while TextEncoder/TextDecoder is being built-in in modern browsers.
#Support
As of July/2018:
TextEncoder (Experimental, On Standard Track)
Chrome | Edge | Firefox | IE | Opera | Safari
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 19° | - | 25 | -
Chrome/A | Edge/mob | Firefox/A | Opera/A |Safari/iOS | Webview/A
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 19° | ? | - | 38
°) 18: Firefox 18 implemented an earlier and slightly different version
of the specification.
WEB WORKER SUPPORT:
Experimental, On Standard Track
Chrome | Edge | Firefox | IE | Opera | Safari
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 20 | - | 25 | -
Chrome/A | Edge/mob | Firefox/A | Opera/A |Safari/iOS | Webview/A
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 20 | ? | - | 38
Data from MDN - `npm i -g mdncomp` by epistemex
Answer from user1693593 on Stack Overflowjavascript - Converting between strings and ArrayBuffers - Stack Overflow
JS: How to convert ArrayBuffer to string?
javascript - Create ArrayBuffer from Array (holding integers) and back again - Stack Overflow
node.js - How to convert a Javascript Object to a Node Buffer? - Stack Overflow
Update 2016 - five years on there are now new methods in the specs (see support below) to convert between strings and typed arrays using proper encoding.
##TextEncoder
The TextEncoder represents:
The
TextEncoderinterface represents an encoder for a specific method, that is a specific character encoding, liarraybufferkeutf-8,An encoder takes a stream of code points as input and emits a stream of bytes.iso-8859-2,koi8,cp1261,gbk, ...
Change note since the above was written: (ibid.)
Note: Firefox, Chrome and Opera used to have support for encoding types other than utf-8 (such as utf-16, iso-8859-2, koi8, cp1261, and gbk). As of Firefox 48 [...], Chrome 54 [...] and Opera 41, no other encoding types are available other than utf-8, in order to match the spec.*
*) Updated specs (W3) and here (whatwg).
After creating an instance of the TextEncoder it will take a string and encode it using a given encoding parameter:
if (!("TextEncoder" in window))
alert("Sorry, this browser does not support TextEncoder...");
var enc = new TextEncoder(); // always utf-8
console.log(enc.encode("This is a a string to be converted to a Uint8Array"));
You then of course use the .buffer parameter on the resulting Uint8Array to convert the underlaying ArrayBuffer to a different view if needed.
Just make sure that the characters in the string adhere to the encoding schema, for example, if you use characters outside the UTF-8 range in the example they will be encoded to two bytes instead of one.
For general use you would use UTF-16 encoding for things like localStorage.
##TextDecoder
Likewise, the opposite process uses the TextDecoder:
The
TextDecoderinterface represents a decoder for a specific method, that is a specific character encoding, likeutf-8,iso-8859-2,koi8,cp1261,gbk, ... A decoder takes a stream of bytes as input and emits a stream of code points.
All available decoding types can be found here.
if (!("TextDecoder" in window))
alert("Sorry, this browser does not support TextDecoder...");
var enc = new TextDecoder("utf-8");
var arr = new Uint8Array([84,104,105,115,32,105,115,32,97,32,85,105,110,116,
56,65,114,114,97,121,32,99,111,110,118,101,114,116,
101,100,32,116,111,32,97,32,115,116,114,105,110,103]);
console.log(enc.decode(arr));
##The MDN StringView library
An alternative to these is to use the StringView library (licensed as lgpl-3.0) which goal is:
- to create a C-like interface for strings (i.e., an array of character codes — an ArrayBufferView in JavaScript) based upon the JavaScript ArrayBuffer interface
- to create a highly extensible library that anyone can extend by adding methods to the object StringView.prototype
- to create a collection of methods for such string-like objects (since now: stringViews) which work strictly on arrays of numbers rather than on creating new immutable JavaScript strings
- to work with Unicode encodings other than JavaScript's default UTF-16 DOMStrings
giving much more flexibility. However, it would require us to link to or embed this library while TextEncoder/TextDecoder is being built-in in modern browsers.
#Support
As of July/2018:
TextEncoder (Experimental, On Standard Track)
Chrome | Edge | Firefox | IE | Opera | Safari
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 19° | - | 25 | -
Chrome/A | Edge/mob | Firefox/A | Opera/A |Safari/iOS | Webview/A
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 19° | ? | - | 38
°) 18: Firefox 18 implemented an earlier and slightly different version
of the specification.
WEB WORKER SUPPORT:
Experimental, On Standard Track
Chrome | Edge | Firefox | IE | Opera | Safari
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 20 | - | 25 | -
Chrome/A | Edge/mob | Firefox/A | Opera/A |Safari/iOS | Webview/A
----------|-----------|-----------|-----------|-----------|-----------
38 | ? | 20 | ? | - | 38
Data from MDN - `npm i -g mdncomp` by epistemex
Although Dennis and gengkev solutions of using Blob/FileReader work, I wouldn't suggest taking that approach. It is an async approach to a simple problem, and it is much slower than a direct solution. I've made a post in html5rocks with a simpler and (much faster) solution: http://updates.html5rocks.com/2012/06/How-to-convert-ArrayBuffer-to-and-from-String
And the solution is:
function ab2str(buf) {
return String.fromCharCode.apply(null, new Uint16Array(buf));
}
function str2ab(str) {
var buf = new ArrayBuffer(str.length*2); // 2 bytes for each char
var bufView = new Uint16Array(buf);
for (var i=0, strLen=str.length; i<strLen; i++) {
bufView[i] = str.charCodeAt(i);
}
return buf;
}
EDIT:
The Encoding API helps solving the string conversion problem. Check out the response from Jeff Posnik on Html5Rocks.com to the above original article.
Excerpt:
The Encoding API makes it simple to translate between raw bytes and native JavaScript strings, regardless of which of the many standard encodings you need to work with.
<pre id="results"></pre>
<script>
if ('TextDecoder' in window) {
// The local files to be fetched, mapped to the encoding that they're using.
var filesToEncoding = {
'utf8.bin': 'utf-8',
'utf16le.bin': 'utf-16le',
'macintosh.bin': 'macintosh'
};
Object.keys(filesToEncoding).forEach(function(file) {
fetchAndDecode(file, filesToEncoding[file]);
});
} else {
document.querySelector('#results').textContent = 'Your browser does not support the Encoding API.'
}
// Use XHR to fetch `file` and interpret its contents as being encoded with `encoding`.
function fetchAndDecode(file, encoding) {
var xhr = new XMLHttpRequest();
xhr.open('GET', file);
// Using 'arraybuffer' as the responseType ensures that the raw data is returned,
// rather than letting XMLHttpRequest decode the data first.
xhr.responseType = 'arraybuffer';
xhr.onload = function() {
if (this.status == 200) {
// The decode() method takes a DataView as a parameter, which is a wrapper on top of the ArrayBuffer.
var dataView = new DataView(this.response);
// The TextDecoder interface is documented at http://encoding.spec.whatwg.org/#interface-textdecoder
var decoder = new TextDecoder(encoding);
var decodedString = decoder.decode(dataView);
// Add the decoded file's text to the <pre> element on the page.
document.querySelector('#results').textContent += decodedString + '\n';
} else {
console.error('Error while requesting', file, this);
}
};
xhr.send();
}
</script>
Working with Web Push Notifications.
https://developer.mozilla.org/en-US/docs/Web/API/PushSubscription
I got the Push Subscription Object.
I get the keys for p256dh and auth, using PushSubscription.getKey()
Unfortunately, getKey() returns an ArrayBuffer per the docs.
I need to save this subscription object with JSON.stringify(), specifically because I am creating a new object and can't use the toJSON() method which is part of the original Push Subscription Object.
When converting to JSON , ArrayBuffer disappears. So I get an empty object.
Thus, I need to convert the ArrayBuffer to a string before sending to the server.
How do I do this? I saw the Buffer suggested, but thats only available on server side.
Yes, there's a simple way without manually writing a loop (the loop still exists somewhere in background):
new Uint16Array([1,2,3]);
That's all. Of course, floating numbers will be rounded down and big numbers will overflow.
Converting typed array to buffer
The buffer of any typed array is accessible through .buffer property, as anyone can read on MDN:
new Uint16Array([1,2,3]).buffer;
Chosing the right typed array
Be warned that mentioned Uint16Array will only hold integers (no floating point) between zero and 65535. To hold any javascript Number1 you will want to use Float64Array - the bigest one, taking 8 bytes total.
1: Which is unrestricted double, which appears to be 64bit IEEE 754 number
Here's a map I have created that maps some of the important information related to number data types:
var NUMBER_TYPE = [
{name: "uint8", bytes:1, max: 255, min: 0, floating: false, array: Uint8Array},
{name: "int8", bytes:1, max: 127, min: -128, floating: false, array: Int8Array},
{name: "uint16", bytes:2, max: 65535, min: 0, floating: false, array: Uint16Array},
{name: "int16", bytes:2, max: 32767, min: -32768, floating: false, array: Int16Array},
{name: "uint32", bytes:4, max: 4294967295, min: 0, floating: false, array: Uint32Array},
{name: "int32", bytes:4, max: 2147483647, min: -2147483648, floating: false, array: Int32Array},
{name: "float64", bytes:8, max: Number.MAX_VALUE, min: Number.MIN_VALUE, floating: true , array: Float64Array}
];
Float 32 is missing as I was unable to calculate necessary information for it. The map, as it is, can be used to calculate the smallest typed array you can fit a Number in:
function findNumberType(num) {
// detect whether number has something after the floating point
var float = num!==(num|0);
// Prepare the return variable
var type = null;
for(var i=0,l=NUMBER_TYPE.length; i<l; i++) {
// Assume this type by default - unless break is hit, every type ends as `float64`
type = NUMBER_TYPE[i];
// Comparison asserts that number is in bounds and disalows floats to be stored
// as integers
if( (!float || type.floating) && num<=type.max && num>=type.min) {
// If this breaks, the smallest data type has been chosen
break;
}
}
return type;
}
Used as:
var n = 1222;
var buffer = new (findNumberType(n).array)([n]);
Note that this only works if NUMBER_TYPE is properly ordered.
You can't use an ArrayBuffer directly, but you can create a typed array from a normal array by using the from method:
let typedArray = Int32Array.from([-2, -1, 0, 1, 2])
A buffer's first argument must be a: String, Buffer, ArrayBuffer, Array, or array-like object.
Taking that information into account, we could implement what you are looking for by creating a buffer from a String. It would look something like the following:
let json = [ 5, false, 55, 'asdf' ];
let buffer = Buffer.from(JSON.stringify(json));
console.log('Buffer: ', buffer); // Buffer: <Buffer 5b 20 35 2c 20 66 61 6c 73 65 2c 20 35 35 2c 20 22 61 73 64 66 22 20 5d>
Then you can bring your JSON back like so:
let converted = JSON.parse(buffer);
console.log('Parsed to json', converted); // Parsed to json [ 5, false, 55, 'asdf' ]
When we are in NodeJS environment we have much better options than, Buffer.from(JSON.stringify(data)).
Performance wise JSON.stringify + Buffer.from() is ok, but will not work out if the object contains ArrayBuffer, and if done, then very inefficient.
Best way for pure NodeJS based environment
Use "v8 Serialization API" Node JS v8 docs
Its easy to use and built into the Node.js binary.
Its fastest and most space efficient among all the serializers of its kind.
const { serialize, deserialize } = require("v8")
const photo = {
name: "rabbit",
height: 1220,
width: 1440,
tinyThumbnail: jpegFileAsBuffer,
mediumThumbnail: anotherJpegFileAsBuffer,
description: "some string data",
metaData: {
tags: ["rabbit", "animal", "runner"],
type: "image/jpeg"
}
}
const photoSerializedAsBuffer = serialize(photo)
const deserialisedBack = deserialize(photo)
But only issue is, this only works for NodeJS. And C++ if you wish to use "v8" library(I personally not a fan of doing that).
For multi platform support
Use "bson" (MongoDB BSON)
Performance wise its close to v8 parser, but it can be adapted in all platforms where MongoDB is supported, NodeJS, JS in web, Java, C++, rust, ruby, python....
The usage is exactly like v8 serialization API
const { serialize, deserialize } = require("bson")
const photo = {
name: "rabbit",
height: 1220,
width: 1440,
tinyThumbnail: jpegFileAsBuffer,
mediumThumbnail: anotherJpegFileAsBuffer,
description: "some string data",
metaData: {
tags: ["rabbit", "animal", "runner"],
type: "image/jpeg"
}
}
const photoSerializedAsBuffer = serialize(photo)
const deserialisedBack = deserialize(photo)
But it can get difficult when BSON types kick in. But this should not be an issue if the object structure is know, and its unlikely that one may not know the object structure while dealing with cross platform stuff.
However a quick, solution for that in NodeJS is to use bson-buffer
At last
This not a full proof solution, but works great for NodeJS and planning to soon launch this for web JS.
tabular-json-node
And due to its simple tabular structure we can support this is other platforms too. Feel free to connect if anyone want to collaborate on this.
» npm install to-array-buffer