I still don't quite understand how they work but after testing a little bit more I found that the following actually worked for what I wanted:

create mask = 0664
force create mode = 0664
directory mask = 0775
force directory mode = 0775

I'll us this for now... Let me know if you have an alternative that makes more sense.

The problem was "obey pam restrictions = yes" was set. Change this to "no" and the permissions are now set correctly. This was a comment in the post Set umask, set permissions, and set ACL, but SAMBA isn't using those?

Often the issue is caused by enabling one of the following:

• map archive (uses unix execute bit for owner)
• map system (uses unix execute bit for group)
• map hidden (uses unix execute bit for other)
• inherit permissions causes samba to ignore create mask, etc
• there's also inherit acls which should default to no

"File Permissions and Attributes on MS-DOS and Unix" in Chapter 8. Advanced Disk Shares explains the above nicely.

However, microsoft office products cause weird behaviour and trigger a bug/unexpected behaviour. Even with create mask and/or force create mode set and avoiding the above map or inherit option, I've seen group execute permissions get set when editing documents using microsoft office applications. This doesn't happen with other programs, e.g. edit a .txt file with notepad and permissions stay sane. Office doesn't just create files, it also messes a bit with temp files, renaming and permissions.

After a lot of digging around, it might be also be mixed up with a Samba POSIX ACL bug of sorts:

• Old 2007 debian bug report "samba: misinterprets create mode with POSIX acls"
• "Wrong behaviour for default ACLs" seems to be the most recent bug report for samba, but is sadly closed 'RESOLVED INVALID' (prematurely?).

Observed behaviour on the Linux side

The extended ACLs for an empty word docx file created via windows explorer context menu (not yet opened and saved by word)

$ getfacl test.docx 
# file: test.docx
# owner: tester
# group: tester
user::rw-
group::rw-
other::---

After saving via MS word 2016

$ getfacl test.docx 
# file: test.docx
# owner: tester
# group: tester
user::rw-
user:tester:rw-
group::rw-
group:tester:rw-
mask::rwx

When checking the samba config, on Ubuntu 16.04 LTS, it seems map archive is enabled by default, but that should use the owner's execute bit, not the group execute bit, so that wasn't the cause in my case

$ testparm -s -v 2>&1 | grep 'map archive'
map archive = Yes

Observed behaviour on the Windows side

Using SysInternals Process Monitor and comparing editing a .txt file with notepad.exe vs. a .docx file with WINWORD.EXE, the office suite does a lot of fancy footwork creating temp files, renaming, etc. In particular, unlike notepad.exe, WINWORD.EXE seems to fiddle with access control lists. As seen by procmon

SetSecurityFile
Information:    Group, DACL

So I suspect this is why the UNIX group permission gets fiddled with by microsoft office applications.

Deeper investigation required turning on windows file object security auditing to see what exactly changed. Event ID 4670 shows the the ~WRD0000.tmp permission changes applied (I think word uses this for periodic saving and recovery if it crashes).

Object:
    Object Server:    Security
    Object Type:    File
    Object Name:    C:\Users\<user>\Desktop\TestAudit\~WRD0000.tmp
    Handle ID:    0xfd0

Process:
    Process ID:    0x1ae4
    Process Name:    C:\Program Files\Microsoft Office\root\Office16\WINWORD.EXE

Permissions Change:
    Original Security Descriptor:
D:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;FA;;;S-1-5-<SID>-1001)
    New Security Descriptor:
D:(A;;0x1200a9;;;WD)(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;FA;;;S-1-5-21-<SID>-1001)

The permission change in SDDL (Security Descriptor Definition Language) is a DACL (discretionary access control list part) where it appears an extra ACE (Access Control Entry) was added. ACE (A;;0x1200a9;;;WD) gets added right at the front. This is how I interpreted it:

• A; meaning allow access, stayed the same
• ; no inheritance
• 0x1200a9; means FILE_GENERIC_READ | FILE_EXECUTE read & execute access
• WD means the 'everyone' trustee

Then later on, the ~WRD0000.tmp gets renamed to test.docx which I saw in process monitor as a SetRenameInformationFile operation. So in the end test.docx ends up with some of the ACE changes word did for whatever reason.

Edit the samba configuration file.

sudo nano /etc/samba/smb.conf

Add this to the end of the file:

[Public Directory]
comment = Contents are read/write by all.
path = /home/user/directory-to-be-shared
read only = no
guest ok = yes
create mask = 0666
force create mode = 0666
directory mask = 0777
force directory mode = 0777

Then restart samba.

sudo service smbd restart

Notes:

1. The string within brackets (Public Directory in the example above) is what users will see when accessing the shared folder from other systems.
2. Files created within this directory by users on other systems will be owned by nobody:nogroup, with read and write permissions for owner, group, and other.
3. Directories created in this directory by users on other systems will be owned by nobody:nogroup, with read, write, and directory traversal permissions for owner, group, and other.
4. The masks and modes in the configuration file apply to files and directories created by users on other systems. These masks and modes are ignored when creating files as a user on the server. Or, to put it another way, if you are user on the server, make sure that each file and directory that you create within /home/user/directory-to-be-shared has 666 and 777 permissions respectively.

5. If you have enabled a firewall on the server, you will need to open port 445 for tcp. For example, if you use ufw (uncomplicated firewall) to configure the firewall:

   sudo ufw allow in 445/tcp
   sudo ufw reload
   sudo ufw status verbose
   

I think you need to use the following parameters:

# I changes the permissions to rw-rw-r--
# You should be able to change them to 775 if you need the files to
# be executable
create mask = 664
force create mode = 664
security mask = 664
force security mode = 664

# I set the SGID flag here as I thought this is what you wanted
# You could change to 0775
directory mask = 2775
force directory mode = 2775
directory security mask = 2775
force directory security mode = 2775

I was looking for a nice explanation of how these settings work, but could not find anything better then man smb.conf

You will have to scroll down a bit for those options.

Basically, in a nutshell, windows permissions are not the same as unix (linux) and it is a bit odd how samba maps permissions.

for what it is worth, I couldn't access my samba share, maybe you have forgotten to add you username to the samba password group (for lack of beter description phrase)

this is what I did to get mine to work

smbpasswd -a username

after by using nautilus, shared my drive/folder with right-click, sharing, etc.

create your password, and use your username and passsword (which you just entered) to access your samba drive, you can also map this drive in Windows

hope this could help