There is no right combination of settings.

The Google Photos upload checkbox (in the Backup and Sync preferences) is a blanket toggle: it affects all the selected local backup/sync folders. So if you want to backup Drive-only files, you need to leave it unchecked and continue manually uploading your images to the Photos app.

As of yet, the B&S client does not allow fine-grained control of what gets uploaded into Photos. Also for backups, be reminded that Photos has limitations on file types and min/max sizes.

Workaround

As a workaround for periodic backups to Photos, you can try the following:

1. (Make sure you don't have Drive and Photos synced with any of the 2 separate settings you mentioned)
2. Open Backup & Sync Preferences → [your device]

3. For "Removing item", select "Don't remove items everywhere".

   

4. Select only the folder(s) you want to back up to Photos

5. Select the "Upload newly added photos [...]" checkbox under the Google Photos Heading. Wait for everything to back up.
6. Uncheck the checkbox under Google Photos again.
7. Uncheck your images folder and re-check your Drive-only folders.

As a side-effect to using B&S, you also retain the images in your Google Drive. But they are linked invisibly to Photos, so:

• they don't eat drive quota (if HQ setting was chosen), and
• deleting them will also remove them from your Photos library.