It seems like you did it backwards and missed a few steps. Here are the full instructions from start to finish.

In Google Domains:

1. Click My domains in the navbar and then click the domain name.

2. Click the DNS tab .

3. Scroll down to Custom resource records.
4. Create a new CNAME record:
5. Name: The subdomain you are directing to the site (do not enter "@").
   • Type: CNAME
   • TTL: 1h
   • Data: GHS.GOOGLEHOSTED.COM
6. Click Add to save the record.

In Google Sites:

1. Log in to Google Sites, and go to the site you are integrating with your domain.
2. Click on the More Actions menu in the top right corner.
3. Select Manage Site.
4. On the left side of the Manage Site page, click Web Address.
5. Under Add a web address enter the address of the subdomain you are using, for example examplesite.documentation.example.
6. Click Add at the top of the page to add your address. Your address should display in a list below the Google Sites address.

IF YOU STILL GET ERRORS:

The reason you get this message is that, for example, when you sign up for Google Apps, we automatically create a site at http://www.example.com mapped to http://sites.google.com/a/example.com/www. When you try to map http://sites.google.com/site/mysite/ to http://www.example.com, this doesn't work, because we've already mapped it for you.

Here's how you can solve this problem:

1. Sign in to the control panel at www.google.com/a/example.com.
2. Click the Settings tab and then select Sites in the left column.
3. Click the Web Address Mapping tab, where you'll see the www mapping. Select the checkbox and click the Delete mapping button.
4. Click Domain Settings tab.
5. Click the Domain names tab.
6. Under the Primary Domain section, click the Advanced DNS Settings link, and read instructions on how to sign in to the DNS registrar's website (either GoDaddy or eNom).
7. Sign in to the DNS registrar, and map the CNAME of www to ghs.google.com.
8. Try to set the web address mapping of http://sites.google.com/site/mysite/ to http://www.example.com.