String::length is a method reference. Using it is equivalent to

Comparator.comparing(s -> s.length())

So it compares strings by comparing their length.

A simple solution is to iterate over all names and compare the length using the .length() method.

Another solution is to compare the length each time you enter a string (only one for loop). Here's an example :

int nb = 2;
String[] names = new String[nb];
int maxLength = -1; // Default value
for (int i = 0; i < nb; i++) {
    names[i] = JOptionPane.showInputDialog("Enter your name : ");
    // Compare the length of this name to the previous greatest length
    if (names[i].length() > maxLength) {
        // Change the maximum
        maxLength = names[i].length();
    }
}
JOptionPane.showMessageDialog(null, "The greatest value is " + maxLength);

The compareTo() method compares two strings lexicographically.

The comparison is based on the Unicode value of each character in the strings.

You need to call length() method for each string to compare lengths

if(one.length() > two.length()) ...

== tests for reference equality (whether they are the same object).

.equals() tests for value equality (whether they contain the same data).

Objects.equals() checks for null before calling .equals() so you don't have to (available as of JDK7, also available in Guava).

Consequently, if you want to test whether two strings have the same value you will probably want to use Objects.equals().

// These two have the same value
new String("test").equals("test") // --> true 

// ... but they are not the same object
new String("test") == "test" // --> false 

// ... neither are these
new String("test") == new String("test") // --> false 

// ... but these are because literals are interned by 
// the compiler and thus refer to the same object
"test" == "test" // --> true 

// ... string literals are concatenated by the compiler
// and the results are interned.
"test" == "te" + "st" // --> true

// ... but you should really just call Objects.equals()
Objects.equals("test", new String("test")) // --> true
Objects.equals(null, "test") // --> false
Objects.equals(null, null) // --> true

From the Java Language Specification JLS 15.21.3. Reference Equality Operators == and !=:

While == may be used to compare references of type String, such an equality test determines whether or not the two operands refer to the same String object. The result is false if the operands are distinct String objects, even if they contain the same sequence of characters (§3.10.5, §3.10.6). The contents of two strings s and t can be tested for equality by the method invocation s.equals(t).

You almost always want to use Objects.equals(). In the rare situation where you know you're dealing with interned strings, you can use ==.

From JLS 3.10.5. String Literals:

Moreover, a string literal always refers to the same instance of class String. This is because string literals - or, more generally, strings that are the values of constant expressions (§15.28) - are "interned" so as to share unique instances, using the method String.intern.

Similar examples can also be found in JLS 3.10.5-1.

Other Methods To Consider

String.equalsIgnoreCase() value equality that ignores case. Beware, however, that this method can have unexpected results in various locale-related cases, see this question.

String.contentEquals() compares the content of the String with the content of any CharSequence (available since Java 1.5). Saves you from having to turn your StringBuffer, etc into a String before doing the equality comparison, but leaves the null checking to you.

From what I understand something like this will work. Remember this will only count unique characters. Order does not matter

public static boolean matchingChar(final String st1, final String st2) {

        if(st1 == null || st2 == null || st1.length() < 5  || st2.length() < 5) {
            return false;
        } 

        //This is if you wish unique characters to be counted only
        //Otherwise you can use simple int count = 0
        HashSet<Character> found = new HashSet<Character>();

        //found.size() < 5 so the loop break as soon as the condition is met
        for(int i = 0; i < st1.length() && found.size() < 5; i++) {         
            if(st2.indexOf(st1.charAt(i)) != -1) {
                found.add(st1.charAt(i));
            }
        }

        return found.size() >= 5;
    }

You need to specify a type parameter for Comparator for your implementation to work.

class comp implements Comparator<String> {
  public int compare(String o1, String o2) {
    if (o1.length() > o2.length()) {
      return 1;
    } else if (o1.length() < o2.length()) {
      return -1;
    } else {
      return 0;
    }
  }
}

In Java 1.7 and later, you can also simplify the body of this method to:

class comp implements Comparator<String> {
  public int compare(String o1, String o2) {
    return Integer.compare(o1.length(), o2.length());
  }
}

Also, Collections.sort sorts List objects. Since you're sorting an array, you should use Arrays.sort:

Arrays.sort(array, new comp());

Show your teacher this!

Stream.generate(scanner::next).limit(5)
    .max(Comparator.comparing(String::length))
    .ifPresent(System.out::println);