Okay, thanks to the people who pointed out the capabilities system and CAP_NET_BIND_SERVICE capability. If you have a recent kernel, it is indeed possible to use this to start a service as non-root but bind low ports. The short answer is that you do:

Copysetcap 'cap_net_bind_service=+ep' /path/to/program

And then anytime program is executed thereafter it will have the CAP_NET_BIND_SERVICE capability. setcap is in the debian package libcap2-bin.

Now for the caveats:

1. You will need at least a 2.6.24 kernel
2. This won't work if your file is a script. (i.e. uses a #! line to launch an interpreter). In this case, as far I as understand, you'd have to apply the capability to the interpreter executable itself, which of course is a security nightmare, since any program using that interpreter will have the capability. I wasn't able to find any clean, easy way to work around this problem.
3. Linux will disable LD_LIBRARY_PATH on any program that has elevated privileges like setcap or suid. So if your program uses its own .../lib/, you might have to look into another option like port forwarding.

Resources:

• capabilities(7) man page. Read this long and hard if you're going to use capabilities in a production environment. There are some really tricky details of how capabilities are inherited across exec() calls that are detailed here.
• setcap man page
• "Bind ports below 1024 without root on GNU/Linux": The document that first pointed me towards setcap.

Note: RHEL first added this in v6.

Until you asked the question I never even heard of this facility in Unix (file capabilities). I found this link which looks to have the solution as to how to make ld.so trust your shared libraries:

• JDK-7157699 : can not run java after granting posix capabilities

excerpt from that post

When one is raising the privileges of an executable, the runtime loader (rtld), better know as ld.so will not link with libraries in untrusted paths. This is the way the ld.so(1) has been designed. If one needs to run such an executable, then you have to add that path to the trusted paths of ld.so, the following describes how to do so:

Fedora 11:
% uname -a
Linux localhost.localdomain 2.6.29.4-167.fc11.i686.PAE #1 SMP Wed May 27 17:28:22 EDT 2009 i686 i686 i386 GNU/Linux

% sudo setcap cap_net_raw+epi ./jdk1.7.0_04/bin/java

% ./jdk1.7.0_04/bin/java -version
./jdk1.7.0_04/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Its kaput, Ok we are on the same page now, to fix this, create a file such as > this, with the path to libjli.so

% cat /etc/ld.so.conf.d/java.conf
/home/someuser/jdk1.7.0_04/jre/lib/i386/jli

This will add the the pathname to the trusted user path, that ld.so will use, to build its runtime cache, verify if ld.so is seeing it by doing this, need to run it as root, and a reboot may be necessary.

% ldconfig | grep libjli
libjli.so -> libjli.so
.......

Now test java:

% ./jdk1.7.0_04/bin/java -version
java version "1.7.0_04-ea"
Java(TM) SE Runtime Environment (build 1.7.0_04-ea-b18)

and there you have it.....

References

• POSIX file capabilities: Parceling the power of root
• This is the Linux kernel capabilities FAQ
• capabilities man page
• Capability-based security - wikipedia
• Using File Capabilities Instead Of Setuid

I'm not sure what the other answers and comments here are referring to. This is possible rather easily. There are two options, both which allow access to low-numbered ports without having to elevate the process to root:

Option 1: Use CAP_NET_BIND_SERVICE to grant low-numbered port access to a process:

With this you can grant permanent access to a specific binary to bind to low-numbered ports via the setcap command:

sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/binary

For more details on the e/i/p part, see cap_from_text.

After doing this, /path/to/binary will be able to bind to low-numbered ports. Note that you must use setcap on the binary itself rather than a symlink.

Option 2: Use authbind to grant one-time access, with finer user/group/port control:

The authbind (man page) tool exists precisely for this.

1. Install authbind using your favorite package manager.

2. Configure it to grant access to the relevant ports, e.g. to allow 80 and 443 from all users and groups:

    sudo touch /etc/authbind/byport/80
    sudo touch /etc/authbind/byport/443
    sudo chmod 777 /etc/authbind/byport/80
    sudo chmod 777 /etc/authbind/byport/443
   

3. Now execute your command via authbind (optionally specifying --deep or other arguments, see the man page):

    authbind --deep /path/to/binary command line args
   

   E.g.

    authbind --deep java -jar SomeServer.jar
   

There are upsides and downsides to both of the above. Option 1 grants trust to the binary but provides no control over per-port access. Option 2 grants trust to the user/group and provides control over per-port access but older versions supported only IPv4 (since I originally wrote this, newer versions with IPv6 support were released).

I think setcap will be your answer. I think the real question here is: How will my system recognize when the webserver on the NFS has been touched so it can run a setcap command on it?

I think you'll want to set up an inotify or systemd.path to monitor this webserver. When that binary gets replaced, you'll detect it and trigger the setcap command that works for you. This works especially well if your webserver runs via systemd already.

Here's an example with systemd.path assuming your server runs as systemd service webserver.service

# /etc/systemd/system/webcap.path
[Unit]
Description=Watching changes in the webserver binary
# Start monitoring only after the webserver is running.
After=webserver.service

[Path]
# Whenever someone writes to this path (binary is replaced), do something
PathModified=/path/to/webserver
# This is the service you launch when the above condition is met
Unit=webcap.service

[Install]
#Whenever the webserver is started, this monitor will also start
WantedBy=webserver.service

# /etc/systemd/system/webcap.service
[Unit]
Description=Update caps of webserver

[Service]
Type=oneshot
ExecStart=ssetcap cap_net_bind_service=ep /path/to/webserver

The way capabilities work in Linux is documented in man 7 capabilities.

Processes' capabilities in the effective set are against which permission checks are done. File capabilities are used during an execv call (which happens when you want to run another program¹) to calculate the new capability sets for the process.

Files have two sets for capabilities, permitted and inheritable and effective bit.

Processes have three capability sets: effective, permitted and inheritable. There is also a bounding set, which limits which capabilities may be added later to a process' inherited set and affects how capabilities are calculated during a call to execv. Capabilities can only be dropped from the bounding set, not added.

Permissions checks for a process are checked against the process' effective set. A process can raise its capabilities from the permitted to the effective set (using capget and capset syscalls, the recommended APIs are respectively cap_get_proc and cap_set_proc).

Inheritable and bounding sets and file capabilities come into play during an execv syscall. During execv, new effective and permitted sets are calculated and the inherited and bounding sets stay unchanged. The algorithm is described in the capabilities man page:

P'(permitted) = (P(inheritable) & F(inheritable)) |
                (F(permitted) & cap_bset)

P'(effective) = F(effective) ? P'(permitted) : 0

P'(inheritable) = P(inheritable)    [i.e., unchanged]

Where P is the old capability set, P' is the capability set after execv and F is the file capability set.

If a capability is in both processes' inheritable set and the file's inheritable set (intersection/logical AND), it is added to the permitted set. The file's permitted set is added (union/logical OR) to it (if it is within the bounding set).

If the effective bit in file capabilities is set, all permitted capabilities are raised to effective after execv.

Capabilities in kernel are actually set for threads, but regarding file capabilities this distinction is usually relevant only if the process alters its own capabilities.

In your example capabilities cap_net_raw , cap_net_admin and cap_dac_override are added to the inherited and permitted sets and the effective bit is set. When your binary is executed, the process will have those capabilities in the effective and permitted sets if they are not limited by a bounding set.

[1] For the fork syscall, all the capabilities and the bounding set are copied from parent process. Changes in uid also have their own semantics for how capabilities are set in the effective and permitted sets.

The reason Solution 1: using a systemd socket doesn't work is that the binary must be built to accept the socket from systemd. It doesn't "just work" unfortunately.

Systemd passes the socket as file descriptor 3 (after stdin, stdout, stderr). Here's an example program (from my blog post here)

package main

import (
    "log"
    "net"
    "net/http"
    "os"
    "strconv"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello World!"))
    })

    if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) {
        // systemd run
        f := os.NewFile(3, "from systemd")
        l, err := net.FileListener(f)
        if err != nil {
            log.Fatal(err)
        }
        http.Serve(l, nil)
    } else {
        // manual run
        log.Fatal(http.ListenAndServe(":8080", nil))
    }
}

You would have to ask goldfish to add support.

For the systemd capabilities (Solution 2) I don't know, but could it be that you also need SecureBits=keep-caps ? systemd should set that automatically, but maybe your version of systemd (8 months ago) didn't? Vault uses it: https://learn.hashicorp.com/vault/operations/ops-deployment-guide#step-3-configure-systemd

Another option would be to use SELinux, although that might be a "now you have two problems".

Personally in your situation I think I would put nginx in front of it. What did you end up doing?