int is a primitive, null is not a value that it can take on. You could change the method return type to return java.lang.Integer and then you can return null, and existing code that returns int will get autoboxed.

Nulls are assigned only to reference types, it means the reference doesn't point to anything. Primitives are not reference types, they are values, so they are never set to null.

Using the object wrapper java.lang.Integer as the return value means you are passing back an Object and the object reference can be null.

StackOverflow has a good discussion about this exact topic in this Q&A. In the top rated question, kronoz notes:

Returning null is usually the best idea if you intend to indicate that no data is available.

An empty object implies data has been returned, whereas returning null clearly indicates that nothing has been returned.

Additionally, returning a null will result in a null exception if you attempt to access members in the object, which can be useful for highlighting buggy code - attempting to access a member of nothing makes no sense. Accessing members of an empty object will not fail meaning bugs can go undiscovered.

Personally, I like to return empty strings for functions that return strings to minimize the amount of error handling that needs to be put in place. However, you'll need to make sure that the group that your working with will follow the same convention - otherwise the benefits of this decision won't be achieved.

However, as the poster in the SO answer noted, nulls should probably be returned if an object is expected so that there is no doubt about whether data is being returned.

In the end, there's no single best way of doing things. Building a team consensus will ultimately drive your team's best practices.

The rationale behind not returning null is that you do not have to check for it and hence your code does not need to follow a different path based on the return value. You might want to check out the Null Object Pattern which provides more information on this.

For example, if I were to define a method in Java that returned a Collection I would typically prefer to return an empty collection (i.e. Collections.emptyList()) rather than null as it means my client code is cleaner; e.g.

Collection<? extends Item> c = getItems(); // Will never return null.

for (Item item : c) { // Will not enter the loop if c is empty.
  // Process item.
}

... which is cleaner than:

Collection<? extends Item> c = getItems(); // Could potentially return null.

// Two possible code paths now so harder to test.
if (c != null) {
  for (Item item : c) {
    // Process item.
  }
}

null is actually not instanceof anything!

The instanceof operator from the Java Language Specification (§15.20.2):

At run time, the result of the instanceof operator is true if the value of the RelationalExpression is not null and the reference could be cast (§15.16) to the ReferenceType without raising a ClassCastException. Otherwise the result is false.

4.1. The Kinds of Types and Values

There are two kinds of types in the Java programming language: primitive types (§4.2) and reference types (§4.3). There are, correspondingly, two kinds of data values that can be stored in variables, passed as arguments, returned by methods, and operated on: primitive values (§4.2) and reference values (§4.3).

Type: PrimitiveType ReferenceType There is also a special null type, the type of the expression null (§3.10.7, §15.8.1), which has no name.

Because the null type has no name, it is impossible to declare a variable of the null type or to cast to the null type.

The null reference is the only possible value of an expression of null type.

The null reference can always undergo a widening reference conversion to any reference type.

In practice, the programmer can ignore the null type and just pretend that null is merely a special literal that can be of any reference type.

In Java, null is only a valid value for reference types. It cannot represent a primitive type such as int. Here are some alternatives to consider:

1. If you are using Java 8 or later, and are able to change the return type of the method you could use the OptionalInt type to represent an int value that may or may not be present. In that case, you would return OptionalInt.empty() in the case that there is no int value to return, and OptionalInt.of(x) to return an int x. Note that the caller will need to unwrap the int value (if it is present) using one of the other methods available on that class. This approach is often preferred for new code, as it makes the intention and usage very clear.
2. If you are using an older Java version, another possibility is to change the return type to Integer. This is a wrapper class for int values that does allow for null to be returned. In addition, Java's auto-unboxing rules allow it to be used in contexts where an int value is expected, when the value is not null. However, if a null value is unboxed to an int value, it will result in a NullPointerException, so it is important to check for null before performing operations that would result in unboxing.
3. If you need to use the int return type, it is common to use a sentinal value to represent an abnormal return. For example, if the normal return values for the method are all non-negative integers, you could use a value such as -1 to represent an absent return value. This is commonly used in older JDK methods such as String.indexOf().
4. In some cases, it makes sense to throw an Exception when no valid value can be returned. It's only a good idea to use this approach for truly exceptional circumstances, as the runtime cost for throwing exceptions is much higher than normal method returns, and the flow of control can make the code harder to understand.