In addition to String.format, also take a look java.text.MessageFormat. The format less terse and a bit closer to the C# example you've provided and you can use it for parsing as well.

For example:

int someNumber = 42;
String someString = "foobar";
Object[] args = {new Long(someNumber), someString};
MessageFormat fmt = new MessageFormat("String is \"{1}\", number is {0}.");
System.out.println(fmt.format(args));

A nicer example takes advantage of the varargs and autoboxing improvements in Java 1.5 and turns the above into a one-liner:

MessageFormat.format("String is \"{1}\", number is {0}.", 42, "foobar");

MessageFormat is a little bit nicer for doing i18nized plurals with the choice modifier. To specify a message that correctly uses the singular form when a variable is 1 and plural otherwise, you can do something like this:

String formatString = "there were {0} {0,choice,0#objects|1#object|1<objects}";
MessageFormat fmt = new MessageFormat(formatString);
fmt.format(new Object[] { new Long(numberOfObjects) });