This is possible since Click 4.0. You need to tell click to ignore the unknown option, and that you want the options gathered up and passed in. Here is your example with the relevant changes made.

Code:

@click.group()
@click.option('--arg')
def cli(arg):
    click.echo('cli arg: %s' % arg)
    pass

@cli.command(context_settings=dict(
    ignore_unknown_options=True,
    help_option_names=[],
))
@click.argument('script')
@click.argument('args', nargs=-1, type=click.UNPROCESSED)
def run(script, args):
    click.echo('script: %s' % script)
    click.echo(args)

Test:

cli('--arg 3 run foo.py --arg 5 arg1 --arg2 --arg 4 --help'.split())

Produces:

cli arg: 3
script: foo.py
('--arg', '5', 'arg1', '--arg2', '"--arg', '4"', '--help')

(Source)

One way that I have managed to achieve this behaviour was by actually using arguments as below. I'll post this as a workaround, while I try to see if it could be done with an option, and I'll update my post accordingly

@click.command(context_settings={"ignore_unknown_options": True})
@click.argument("options", nargs=-1)
def myscript(options):
    option = False
    if options is ():
        option = False
    if '-o' in options or '--option' in options:
        option = True
    if len(options) > 1:
        option = options[1]
    print(option)

Later Edit Using an option, I have managed to achieve this by adding an argument to the command definition.

@click.command()
@click.option('-o', '--option', is_flag=True, default=False, is_eager=True)
@click.argument('value', nargs=-1)
def myscript(option, value):
    if option and value != ():
        option = value[0]
    print(option)

The nargs can be removed if you only expect at most one argument to follow, and can be treated as not required.

@click.command()
@click.option('-o', '--option', is_flag=True, default=False, is_eager=True)
@click.argument('value', required=False)
def myscript(option, value=None):
    if option and value is not None:
        option = value
    print(option)

It might also be possible by putting together a context generator and storing some state, but that seems the least desirable solution, since you would be relying on the context storing your state.

This can be done by over riding the click.Group argument parser like:

Custom Class:

class MyGroup(click.Group):
    def parse_args(self, ctx, args):
        if args[0] in self.commands:
            if len(args) == 1 or args[1] not in self.commands:
                args.insert(0, '')
        super(MyGroup, self).parse_args(ctx, args)

Using Custom Class:

Then to use the custom group, pass it as the cls argument to the group decorator like:

@click.group(cls=MyGroup)
@click.argument('hostname', required=False)
@click.pass_context
def cli(ctx, hostname=None):
    ....

How?

This works because click is a well designed OO framework. The @click.group() decorator usually instantiates a click.Group object but allows this behavior to be over ridden with the cls parameter. So it is a relatively easy matter to inherit from click.Group in our own class and over ride desired methods.

In this case we over ride click.Group.parse_args() and if the first parameter matches a command and the second parameter does not, then we insert an empty string as the first parameter. This puts everything back where the parser expects it to be.

The simplest solution for single arguments or options seems to be to just create a variable ARG = click.option(...) and then use @ARG as a decorator of the command function.

Applying multiple opts/args at once involves creating a decorator factory or a callable class. This is the cleanest solution I found, credits go to @altendky and @SnoopJ from #python channel on Libera IRC:

from typing import List, Callable

import attr
import click

from click.decorators import FC

options = [
    click.option("-a"),
    click.option("-b"),
]


# Solution 1: decorator factory
def multi_decorator(options: List[Callable[[FC], FC]]) -> Callable[[FC], FC]:
    def decorator(f: FC) -> FC:
        for option in reversed(options):
            f = option(f)

        return f

    return decorator


@click.command()
@multi_decorator(options)
def cmd1(a, b):
    pass


# Solution 2: callable class
@attr.s
class MultiDecorator:
    options: List[Callable[[FC], FC]] = attr.ib(factory=list)

    def __call__(self, f: FC):
        for option in reversed(self.options):
            f = option(f)

        return f


@click.command()
@MultiDecorator(options)
def cmd2(a, b):
    pass


if __name__ == "__main__":
    from click.testing import CliRunner
    runner = CliRunner()

    print(runner.invoke(cmd1, ["--help"]).output)
    print(runner.invoke(cmd2, ["--help"]).output)

If you want to test or improve this, head over to https://replit.com/@ldericher/multidecorator.

If you don't insist on passing something that looks like a list, but simply want to pass multiple variadic arguments, you can use the multiple option.

From the click documentation

@click.command()
@click.option('--message', '-m', multiple=True)
def commit(message):
    click.echo('\n'.join(message))

$ commit -m foo -m bar
foo
bar

import sys
import click
import pandas as pd
from sklearn import metrics
@click.command()

# @click.option('--filename')
## @click.argument('result.csv', type=click.Path(exists=True))
@click.argument('filename', type=click.Path(exists=True))
def main(filename):

    df = pd.read_csv(filename)
    y_test = df["actual"].values
    y_pred = df["predicted"].values

    print('R Squared: ' +  str(metrics.r2_score(y_test, y_pred)))
    
if __name__ == '__main__':
    # sys.argv = ['',  '--filename', 'result.csv']
    sys.argv = ['',  'result.csv']
    main()

As it turns out I misunderstood the usage of multiple=True when using @click.option.

For example is it possible to invoke -t multiple times

python selectTools.py -t tool1 -t tool3 -t tool2

by using

@click.option('--tools', '-t', type=click.Choice(['tool1', 'tool2', 'tool3']), multiple=True)

and thus makes the selection of several tools from the choices possible.

The Forwarding Unknown Options section in the documentation covers this use case. You should be able to write something like this:

@click.command(context_settings=dict(
    ignore_unknown_options=True,
))
@click.option("--port", type=int)
@click.argument("args", nargs=-1, type=click.UNPROCESSED)
def main(port, args):
    call_the_wrapped_program(port=port, args=args)

This can be done by over riding the click.Command.invoke() method like:

Custom Class:

def CommandWithConfigFile(config_file_param_name):

    class CustomCommandClass(click.Command):

        def invoke(self, ctx):
            config_file = ctx.params[config_file_param_name]
            if config_file is not None:
                with open(config_file) as f:
                    config_data = yaml.safe_load(f)
                    for param, value in ctx.params.items():
                        if value is None and param in config_data:
                            ctx.params[param] = config_data[param]

            return super(CustomCommandClass, self).invoke(ctx)

    return CustomCommandClass

Using Custom Class:

Then to use the custom class, pass it as the cls argument to the command decorator like:

@click.command(cls=CommandWithConfigFile('config_file'))
@click.argument("arg")
@click.option("--opt")
@click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):

Test Code:

# !/usr/bin/env python
import click
import yaml

@click.command(cls=CommandWithConfigFile('config_file'))
@click.argument("arg")
@click.option("--opt")
@click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
    print("arg: {}".format(arg))
    print("opt: {}".format(opt))
    print("config_file: {}".format(config_file))


main('my_arg --config_file config_file'.split())

Test Results:

arg: my_arg
opt: my_opt
config_file: config_file