Update 3/2022

There is currently a proposal to add Records and Tuples (basically immutable Objects and Arrays) to Javascript. In that proposal, it offers direct comparison of Records and Tuples using === or !== where it compares values, not just object references AND relevant to this answer both Set and Map objects would use the value of the Record or Tuple in key comparisons/lookups which would solve what is being asked for here.

Since the Records and Tuples are immutable (can't be modified) and because they are easily compared by value (by their contents, not just their object reference), it allows Maps and Sets to use object contents as keys and the proposed spec explicitly names this feature for Sets and Maps.

This original question asked for customizability of a Set comparison in order to support deep object comparison. This doesn't propose customizability of the Set comparison, but it directly supports deep object comparison if you use the new Record or a Tuple instead of an Object or an Array and thus would solve the original problem here.

Note, this proposal advanced to Stage 2 in mid-2021. It has been moving forward recently, but is certainly not done.

Mozilla work on this new proposal can be tracked here.

Official Spec Draft here.

Incomplete polyfill here.

Note the polyfill will never be a complete polyfill because the spec uses new language features and implements new types in the language. But, the polyfill can be used with some work-arounds.

Original Answer

The ES6 Set object does not have any compare methods or custom compare extensibility.

The .has(), .add() and .delete() methods work only off it being the same actual object or same value for a primitive and don't have a means to plug into or replace just that logic.

You could presumably derive your own object from a Set and replace .has(), .add() and .delete() methods with something that did a deep object comparison first to find if the item is already in the Set, but the performance would likely not be good since the underlying Set object would not be helping at all. You'd probably have to just do a brute force iteration through all existing objects to find a match using your own custom compare before calling the original .add().

Here's some info from this article and discussion of ES6 features:

5.2 Why can’t I configure how maps and sets compare keys and values?

Question: It would be nice if there were a way to configure what map keys and what set elements are considered equal. Why isn’t there?

Answer: That feature has been postponed, as it is difficult to implement properly and efficiently. One option is to hand callbacks to collections that specify equality.

Another option, available in Java, is to specify equality via a method that object implement (equals() in Java). However, this approach is problematic for mutable objects: In general, if an object changes, its “location” inside a collection has to change, as well. But that’s not what happens in Java. JavaScript will probably go the safer route of only enabling comparison by value for special immutable objects (so-called value objects). Comparison by value means that two values are considered equal if their contents are equal. Primitive values are compared by value in JavaScript.

Try this:

const eqSet = (xs, ys) =>
    xs.size === ys.size &&
    [...xs].every((x) => ys.has(x));

const ws = new Set([1, 2, 3]);
const xs = new Set([1, 3, 2]);
const ys = new Set([1, 2, 4]);
const zs = new Set([1, 2, 3, 4]);

console.log(eqSet(ws, xs)); // true
console.log(eqSet(ws, ys)); // false
console.log(eqSet(ws, zs)); // false

Assuming the provided objects don't contain values that cannot be stringified to JSON (function, undefined, symbol, etc.) You can use JSON.stringify().

One problem you might encounter is that stringifying { a: 1, b: 2 } doesn't produce the same result as { b: 2, a: 1 }. A fairly easy way to solve this would be to stringify the object and make sure the resulting JSON has properties placed in alphabetical order.

For this we can look to the answer provided in sort object properties and JSON.stringify.

I also think you are over complicating things by only stringifying values if they are an object. Instead you could just stringfy everything, null would result in "null", "string" would result in '"string"', etc. This simplifies the code by a lot. The only restriction then becomes that all values must be a valid JSON value.

// see linked answer
function JSONstringifyOrder(obj, space)
{
    const allKeys = new Set();
    JSON.stringify(obj, (key, value) => (allKeys.add(key), value));
    return JSON.stringify(obj, Array.from(allKeys).sort(), space);
}

class MySet extends Set {
    // The constructor makes uses of add(), so we don't need
    // to override the constructor.

    has(item) {
        return super.has(JSONstringifyOrder(item));
    }
    
    add(item) {
        return super.add(JSONstringifyOrder(item));
    }
    
    delete(item) {
        return super.delete(JSONstringifyOrder(item));
    }
}


const set = new MySet([[1,2,3], [1,2,3], 1, 2, 1]);
set.add([1,2,3]);
set.add({ a: { s: 1, d: 2 }, f: 3 });
set.add({ f: 3, a: { d: 2, s: 1 } });

// Stack Overflow snippets cannot print Set instances to the console
console.log(Array.from(set));
// or unserialized
Array.from(set, json => JSON.parse(json)).forEach(item => console.log(item));