Valuable and invaluable are not antonyms, but the prefix in- still means not. M-W explains:

Valuable and invaluable do mean similar things, but the in- in invaluable isn't playing any tricks. It simply means "not." We think of valuable as meaning "having a great deal of value," as in "valuable jewelry" or "learned a valuable lesson."

Invaluable, on the other hand, means "valuable beyond estimation." Much like priceless, it describes something that is of such a great value that it cannot fairly be quantified.

To understand how invaluable was formed, Etymonline comes handy:

1570s, "above value, too valuable for exact estimate," from in- "not" + value (v.) "estimate the worth of" + -able. It also has been used in a sense "without value, worthless" (1630s, from in- + valuable).

Although it has been used as the antonym of valuable in the past, now that meaning has not survived.

M-W says:

While the verb value often means "to prize or esteem" (as in "I value our friendship"), it can also mean "to estimate or assign the monetary worth of" or "appraise." If something is of such a nature that its importance cannot be stated in monetary terms, that obviously makes it unable to be valued, or invaluable.