Allow me to preface this answer by stating that all of these hooks are very rarely used. 99% of the time, you won't need these. They are only meant to cover some rare corner-case scenarios.

useImperativeHandle

Usually when you use useRef you are given the instance value of the component the ref is attached to. This allows you to interact with the DOM element directly.

useImperativeHandle is very similar, but it lets you do two things:

1. It gives you control over the value that is returned. Instead of returning the instance element, you explicitly state what the return value will be (see snippet below).
2. It allows you to replace native functions (such as blur, focus, etc) with functions of your own, thus allowing side-effects to the normal behavior, or a different behavior altogether. Though, you can call the function whatever you like.

There could be many reasons you want might to do either of the above; you might not want to expose native properties to the parent or maybe you want to change the behavior of a native function. There could be many reasons. However, useImperativeHandle is rarely used.

useImperativeHandle customizes the instance value that is exposed to parent components when using ref

Example

In this example, the value we'll get from the ref will only contain the function blur which we declared in our useImperativeHandle. It will not contain any other properties (I am logging the value to demonstrate this). The function itself is also "customized" to behave differently than what you'd normally expect. Here, it sets document.title and blurs the input when blur is invoked.

const MyInput = React.forwardRef((props, ref) => {
  const [val, setVal] = React.useState('');
  const inputRef = React.useRef();

  React.useImperativeHandle(ref, () => ({
    blur: () => {
      document.title = val;
      inputRef.current.blur();
    }
  }));

  return (
    <input
      ref={inputRef}
      val={val}
      onChange={e => setVal(e.target.value)}
      {...props}
    />
  );
});

const App = () => {
  const ref = React.useRef(null);
  const onBlur = () => {
    console.log(ref.current); // Only contains one property!
    ref.current.blur();
  };

  return <MyInput ref={ref} onBlur={onBlur} />;
};

ReactDOM.render(<App />, document.getElementById("app"));

<script src="https://cdnjs.cloudflare.com/ajax/libs/react/16.8.1/umd/react.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.8.1/umd/react-dom.production.min.js"></script>
<div id="app"></div>

useLayoutEffect

While similar to some extent to useEffect(), it differs in that it will run after React has committed updates to the DOM. Used in rare cases when you need to calculate the distance between elements after an update or do other post-update calculations / side-effects.

The signature is identical to useEffect, but it fires synchronously after all DOM mutations. Use this to read layout from the DOM and synchronously re-render. Updates scheduled inside useLayoutEffect will be flushed synchronously, before the browser has a chance to paint.

Example

Suppose you have an absolutely positioned element whose height might vary and you want to position another div beneath it. You could use getBoundingClientRect() to calculate the parent's height and top properties and then just apply those to the top property of the child.

Here you would want to use useLayoutEffect rather than useEffect. See why in the examples below:

With useEffect: (notice the jumpy behavior)

const Message = ({boxRef, children}) => {
  const msgRef = React.useRef(null);
  React.useEffect(() => {
    const rect = boxRef.current.getBoundingClientRect();
    msgRef.current.style.top = `${rect.height + rect.top}px`;
  }, []);

  return <span ref={msgRef} className="msg">{children}</span>;
};

const App = () => {
  const [show, setShow] = React.useState(false);
  const boxRef = React.useRef(null);

  return (
    <div>
      <div ref={boxRef} className="box" onClick={() => setShow(prev => !prev)}>Click me</div>
      {show && <Message boxRef={boxRef}>Foo bar baz</Message>}
    </div>
  );
};

ReactDOM.render(<App />, document.getElementById("app"));

.box {
  position: absolute;
  width: 100px;
  height: 100px;
  background: green;
  color: white;
}

.msg {
  position: relative;
  border: 1px solid red;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/react/16.8.1/umd/react.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.8.1/umd/react-dom.production.min.js"></script>
<div id="app"></div>

With useLayoutEffect:

const Message = ({boxRef, children}) => {
  const msgRef = React.useRef(null);
  React.useLayoutEffect(() => {
    const rect = boxRef.current.getBoundingClientRect();
    msgRef.current.style.top = `${rect.height + rect.top}px`;
  }, []);

  return <span ref={msgRef} className="msg">{children}</span>;
};

const App = () => {
  const [show, setShow] = React.useState(false);
  const boxRef = React.useRef(null);

  return (
    <div>
      <div ref={boxRef} className="box" onClick={() => setShow(prev => !prev)}>Click me</div>
      {show && <Message boxRef={boxRef}>Foo bar baz</Message>}
    </div>
  );
};

ReactDOM.render(<App />, document.getElementById("app"));

.box {
  position: absolute;
  width: 100px;
  height: 100px;
  background: green;
  color: white;
}

.msg {
  position: relative;
  border: 1px solid red;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/react/16.8.1/umd/react.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.8.1/umd/react-dom.production.min.js"></script>
<div id="app"></div>

useDebugValue

Sometimes you might want to debug certain values or properties, but doing so might require expensive operations which might impact performance.

useDebugValue is only called when the React DevTools are open and the related hook is inspected, preventing any impact on performance.

useDebugValue can be used to display a label for custom hooks in React DevTools.

I have personally never used this hook though. Maybe someone in the comments can give some insight with a good example.

The following diagram always helps me to visualize the flow of hooks and also will let you clarify regarding why useLayoutEffect is recommended over useEffect for DOM based operations (where you're targetting stuff that you can update before current browser paint)

Link to the diagram - https://github.com/donavon/hook-flow