The documentation says:

The || operator concatenates the elements at the top level of each of its operands. ... For example, if both operands are objects with a common key field name, the value of the field in the result will just be the value from the right hand operand.

So using your example data:

update table_name set
    data = data || '{"city": "ottawa", "phone": "phonenum", "prefix": "prefixedName"}'
    where number = 1;

Additionally if the object you want to edit is not at the top level - just combine the concatenation and jsonb_set function. For example, if the original data looks like

{"location": {"name": "firstName", "city": "toronto", "province": "ON"}}

then

...
data = jsonb_set(
    data, 
    '{location}', data->'location' || '{"city": "ottawa", "phone": "phonenum", "prefix": "prefixedName"}')
...

The document says:

jsonb_set(target jsonb, path text[], new_value jsonb[, create_missing boolean])

In your example, the second parameter - '{k2,value}' is the search path, but since the first path is k2 that does not exist, results to NULL before value could be added/replaced.

In simple terms – jsonb_set is not meant to build entire JSON document the way you tried using search path, but to add or replace single key/value.

If you want to add/replace entirely new set of JSON like that you can use || (concatenate) operator instead like that:

-- Add example:
SELECT $${"k1": {"value": "v1"}}$$::jsonb || $${ "k2": { "value": "v2"}}$$::jsonb;
                    ?column?
------------------------------------------------
 {"k1": {"value": "v1"}, "k2": {"value": "v2"}}
(1 row)

-- Replace example
SELECT $${"k1": {"value": "v1"}}$$::jsonb || $${ "k1": { "value": "v2"}}$$::jsonb;
        ?column?
-------------------------
 {"k1": {"value": "v2"}}
(1 row) 

OR

You can use jsonb_set() like that instead:

SELECT JSONB_SET(
    '{"k1": {"value": "v1"}}',
    '{k2}',
    '{"value": "v2"}',
    TRUE
);
                   jsonb_set
------------------------------------------------
 {"k1": {"value": "v1"}, "k2": {"value": "v2"}}
(1 row)