The max call stack exceeded error is being generated by converting the hex string into an array and supplying it to String.fromCharCode() as an arguments array using the apply method from Function.prototype.

There are practical limits on how many arguments can be passed in JavaScript function calls so the method can work for small files and fail for larger ones.

Refactoring the code will be necessary. As an untested basic solution

function hexToBase64(str) {
    var bString = "";
    for( var i = 0; i < str.length; i +=2) {
         bString += String.fromCharCode( parseInt( str.substr( i, 2), 16));
    }
    return btoa(bString);
}

may provide some ideas.

I was able to get this working. Here is how.

In sendHex, I removed the &H portion, and wrapped my string in hex():

Private Sub sendHex(valHex)
    Dim iCntHex
    For iCntHex = 1 To len(valHex) Step 2
    If len( mid(valHex, iCntHex, 2)) = 1 Then 
        response.write "0"
    end if 
    response.write mid(valHex, iCntHex, 2)
    Next
End Sub

This results in a string output like this (in byte strings of 2 hexidecimal chars):

424d1e050000000000003e00000028000000340000001800000001000

I can then dump that proper hex string into a HEX to base64 function as follows (not written by me, but rather, by Richard Mueller - http://www.rlmueller.net/Base64.htm)

Function HexToBase64(strHex)
    ' Function to convert a hex string into a base64 encoded string.
    ' Constant B64 has global scope.
    Dim lngValue, lngTemp, lngChar, intLen, k, j, strWord, str64, intTerm

    intLen = Len(strHex)

    ' Pad with zeros to multiple of 3 bytes.
    intTerm = intLen Mod 6
    If (intTerm = 4) Then
        strHex = strHex & "00"
        intLen = intLen + 2
    End If
    If (intTerm = 2) Then
        strHex = strHex & "0000"
        intLen = intLen + 4
    End If

    ' Parse into groups of 3 hex bytes.
    j = 0
    strWord = ""
    HexToBase64 = ""
    For k = 1 To intLen Step 2
        j = j + 1
        strWord = strWord & Mid(strHex, k, 2)
        If (j = 3) Then
            ' Convert 3 8-bit bytes into 4 6-bit characters.
            lngValue = CCur("&H" & strWord)

            lngTemp = Fix(lngValue / 64)
            lngChar = lngValue - (64 * lngTemp)
            str64 = Mid(B64, lngChar + 1, 1)
            lngValue = lngTemp

            lngTemp = Fix(lngValue / 64)
            lngChar = lngValue - (64 * lngTemp)
            str64 = Mid(B64, lngChar + 1, 1) & str64
            lngValue = lngTemp

            lngTemp = Fix(lngValue / 64)
            lngChar = lngValue - (64 * lngTemp)
            str64 = Mid(B64, lngChar + 1, 1) & str64

            str64 = Mid(B64, lngTemp + 1, 1) & str64

            HexToBase64 = HexToBase64 & str64
            j = 0
            strWord = ""
        End If
    Next
    ' Account for padding.
    If (intTerm = 4) Then
        HexToBase64 = Left(HexToBase64, Len(HexToBase64) - 1) & "="
    End If
    If (intTerm = 2) Then
        HexToBase64 = Left(HexToBase64, Len(HexToBase64) - 2) & "=="
    End If

End Function

This converts the above to base64, and I can use the output like this (e.g. in a browser url bar) to view it as an image:

data:image/bmp;base64,Qk0eBQAAAAAAAD4AAAAo...

Use xxd with the -r argument (and possibly the -p argument) to convert from hex to plain binary/octets and base64 to convert the binary/octet form to base64.

For a file:

cat file.dat | xxd -r -p | base64

For a string of hex numbers:

echo "6F0AD0BFEE7D4B478AFED096E03CD80A" | xxd -r -p | base64

Edit 26 Aug 2020: As suggested by Ali in the comments, using codecs.encode(b, "base64") would result in extra line breaks for MIME syntax. Only use this method if you do want those line breaks formatting.

For a plain Base64 encoding/decoding, use base64.b64encode and base64.b64decode. See the answer from Ali for details.

In Python 3, arbitrary encodings including Hex and Base64 has been moved to codecs module. To get a Base64 str from a hex str:

import codecs

hex = "10000000000002ae"
b64 = codecs.encode(codecs.decode(hex, 'hex'), 'base64').decode()