The solution was simple, all I was missing was the return at the end of the command being sent. I was able to operate the SMAC Controller with the following command:

0x20 W 0x012C04 20\r

The full code used is:

const port = new SerialPort(SMAC, {
    baudRate: 115200,
    dataBits: 8,
    stopBits: 1,
    parity: "none",
}, (err) => console.log(err))

if(!port.isOpen) port.open()

port.on('open', () => {
    console.log("Port opened successfully")

    const macro = "0x20 W 0x012C04 20\r"

    // Call Macro 20
    port.write(macro, (err) => {
        if(err) throw err;
        console.log("Write to port successful")
        port.close()
    })
})

I think I got it figured out from the github (https://github.com/voodootikigod/node-serialport page... basically it looks like I was missing the "open" event as shown below:

serialPort.on("open", function () {
  console.log("open");
  serialPort.on("data", function(data) {
    console.log("data received: " + data);
  });  
  serialPort.write("SYST:ADDR?\n", function(err, results) {
    console.log("err: " + err);
    console.log("results: " + results);
  });  
});

I figured it out. I put a serial port monitoring application on to see what was happening.

The final answer was

A. When using an array or buffer to pass to the .write() method, it needs to be either a decimal character code (ie. 30 for "0" or 13 for carriage return) or a hex code (ie. 0x30 for "0" or 0x0D for carriage return) for each item in the array. I didn't try unicode or anything else. But see below my thoughts on that.

B. It turns out that in my example code

sp.on("open", function () {
    console.log(comPort + ' is open');
    sp.write(0x80);
    sp.write('123456\r');
});

I needed to be sending the 0x80 as a buffer/array instead. so sp.write([0x80]);, without it sent lots of 00 00 00...s However the LED display vendor's documentation was incorrect as it needed a \r\n at the end and not just a \r.

C. (a bit of a "duh" moment) was simply that Javascript will attempt to convert anything concatenated with a string to a string. So concatenating 0x80 + '123456\r' turns into '128123456\r'.

D. Passing unicode, like sp.write('\u0080') ends up outputing double characters, I'm assuming since unicode is 2 bytes per character, and the string is being treated/expecting single byte encoding. Javascript strings are UTF-16, so I'm guessing the node serialport module doesn't handle it since RS-232 uses 8 bits or less per character which falls into the typical ASCII char set.

I found the answer myself!
I was using the wrong baudRate, and also needed to stringify the data being sent as a JSON string

I wrap a promise in the serial data receive

function sendSync(port, src) {
    return new Promise((resolve, reject) => {
        port.write(src);
        port.once('data', (data) => {
            resolve(data.toString());
        });

        port.once('error', (err) => {
            reject(err);
        });
    });
}

Please take note, the event is using once instead of on to prevent event from stacking (please check the comments below for more information - thanks @DKebler for spotting it)

Then, I could write the code in sync as below

sendSync(port, 'AThello\n').then((data) => {
    //receive data
});

sendSync(port, 'ATecho\n').then((data) => {
    //receive data
});

or I could use a generator, using co package

 co(function* () {
        const echo = yield sendSync(port, 'echo\n');
        const hello = yield sendSync(port, 'hello 123\n');

        return [echo, hello]
    }).then((result) => {
        console.log(result)
    }).catch((err) => {
        console.error(err);
    })

Finally I was able to figure it out. Just create a buffer variable (as mentioned in the documentation) and add those bytes to it. Write it to the serial port. Below is the chunk which worked for me:

var buffer = new Buffer(10);
buffer[0] = 0x05;
buffer[1] = 0xAA;
buffer[2] = 0x55;
buffer[3] = 0xFA;
buffer[4] = 0x00;
buffer[5] = 0x56;
buffer[6] = 0x00;
buffer[7] = 0x03;
buffer[8] = 0x9E;
buffer[9] = 0x00;

var com = new SerialPort(COM1, {
    baudRate: 38400,
    databits: 8,
    parity: 'none'
}, false);

com.open(function (error) {
    if (error) {
        console.log('Error while opening the port ' + error);
    } else {
        console.log('CST port open');
        com.write(buffer, function (err, result) {
            if (err) {
                console.log('Error while sending message : ' + err);
            }
            if (result) {
                console.log('Response received after sending message : ' + result);
            }    
        });
    }              
});