Interestingly, it is possible to post from any branch, just not from any job.

To do this, I needed to make two changes:

1. I need to know the current state of the published data
2. I need to change directory based on the current branch

GitLab has the ability to cache folders. Generally this is used to speed up builds by caching downloaded drivers. There is no reasons I could not use this to store the public folder. This way when I make changes to staging, I will remember the state of the root application:

cache:
  paths:
  - public

The next trick would be to publish pages to the appropriate folder, depending on current branch being built. To do this, we can look to GitLab CI/CD Environment Variables; in particular:

• CI_COMMIT_REF_SLUG: The current branch
• CI_DEFAULT_BRANCH: the default branch (master)

Knowing these two values, we can do a bit of bash to determine the correct place to write the content to.

pages:
  stage: deploy
  script:
  - dir="$CI_COMMIT_REF_SLUG"
  - if [ "$CI_COMMIT_REF_SLUG" == "$CI_DEFAULT_BRANCH" ]; then dir=""; fi;
  - dir="public/$dir"
  - echo "Deploying to $dir"
  - mkdir -p dir
  artifacts:
    paths:
    - public
  only:
  - staging
  - master

Don't forget to limit pages to only staging and master.

WARNING

I'm not satisfied with this.

I think it would be better to maintain the cache somewhere completely different and copy them in at a later stage, but completely re-writing the public folder each time.

The current solution will build cruft over time, but the basic idea is sound.

I've had success using the browsable artifacts for this purpose. In your example, you would create a job for your develop branch and set the PUBLIC_URL to the path on gitlab.io where the job's artifacts are published:

develop:
    artifacts:
        paths:
          - public

    environment:
        name: Develop
        url: "https://$CI_PROJECT_NAMESPACE.gitlab.io/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html"

    script: |
        # whatever

    stage: deploy

    variables:
        PUBLIC_URL: "/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public"

Setting the environment as indicated produces a »Review app« link in relevant merge requests, allowing you to get to the artifacts with a single click.

Note: if your repository is in a subgroup, you need to insert the subgroup name in two places above above between /-/ and $CI_PROJECT_NAME for the resulting URLs to work.

Using cache is the right direction, but the solution needs some side features to be fully satisfying.

With below solution, from GitLab Pages per branch : the no-compromise hack to serve preview pages , you can have these features :

• main content is exposed on $CI_PAGES_URL and the path is configurable with $CURRENT_CONTENT_PATH
• Per-branch preview content is exposed on $CI_PAGES_URL/preview, with a homepage to easily navigate to branches content
• Path to root preview folder is configurable with $EPHEMERAL_BRANCHES_PATH variable to hide preview content by obfuscation
• Generated pages are associated with environments to take advantage of auto-cleaning on branch deletion
• To avoid disturbing already existing environments, pages environment are placed under a pages folder
• If main content has not been generated in current cache, or if the cache has been deleted, an error is triggered, to avoid accidental deletion
• Deletion job can be triggered manually with any cache path as input, to clean outdated data
• Code can safely be added to an existing project pipeline without causing trouble with already existing jobs
• The workflow:rules can be deleted if you already have your own, or updated to match your flow
• The job must be named pages and the artifact must be a public folder to be deployed to GitLab Pages (or you can use the pages:publish keyword)

workflow:
  rules: # disable tag pipelines and duplicate MR pipelines
    - if: $CI_COMMIT_BRANCH

variables:
  EPHEMERAL_BRANCHES_PATH: preview # subpath to ephemeral branches content for preview, anything will work

pages:
  stage: build
  image: alpine:3.18
  cache:
    key: gitlab-pages
    paths: [public]
  before_script:
    # default available 'tree' app in alpine image does not work as intended
    - apk add tree
    # CURRENT_CONTENT_PATH is defined in rules, different between main branch and ephemeral branches
    - mkdir -p public/$CURRENT_CONTENT_PATH && ls public/$CURRENT_CONTENT_PATH/..
    - | # avoid deleting main branch content when cache has been erased
      if [ "$CI_COMMIT_BRANCH" != "$CI_DEFAULT_BRANCH" ] && [ ! -d public/$CI_DEFAULT_BRANCH ]; then
        echo -e "💥\e[91;1m Unable to retrieve $CI_DEFAULT_BRANCH generated files from cache ; please regenerate $CI_DEFAULT_BRANCH files first\e[0m"
        exit 1
      fi
    - rm -rf public/$CURRENT_CONTENT_PATH || true # remove last version of current branch
  script:
    - ./generate-my-html.sh --output build-docs || true # insert here your code that generates documentation
    - mv --verbose build-docs public/$CURRENT_CONTENT_PATH
    - cd public/$EPHEMERAL_BRANCHES_PATH
    - tree -d -H '.' -L 1 --noreport --charset utf-8 -T "Versions" -o index.html # generate a root HTML listing all previews for easier access
  environment:
    name: pages/$CI_COMMIT_BRANCH
    action: start
    url: $CI_PAGES_URL/$CURRENT_CONTENT_PATH
    on_stop: pages-clean-preview
  rules:
    # 'main branch' is exposed at GitLab Pages root
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      variables:
        CURRENT_CONTENT_PATH: "."
    # other (short-lived) branches generation are exposed in 'EPHEMERAL_BRANCHES_PATH/branch-name-sanitized' sub path
    - variables:
        CURRENT_CONTENT_PATH: $EPHEMERAL_BRANCHES_PATH/$CI_COMMIT_REF_SLUG
  artifacts:
    paths: [public]
    expire_in: 1h

pages-clean-preview:
  stage: build
  image: alpine:3.18
  cache:
    key: gitlab-pages
    paths: [public]
  variables:
    GIT_STRATEGY: none # git files not available after branch deletion
    FOLDER_TO_DELETE: preview/$CI_COMMIT_BRANCH # an indirection to allow arbirtraty deletion when launching this job
  script:
    - rm -rf public/$FOLDER_TO_DELETE
  environment:
    name: pages/$CI_COMMIT_BRANCH
    action: stop
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true

Only one site is supported on these platforms. You cannot have separate sites for separate branches.

The only way you might do this without some sort of cache/artifact retrieval (as you mentioned is another option in your question) is to build all your branches at once when publishing your Pages site.

How exactly you do that depends on a lot of factors, including what tool(s) you're using to build your site and if they are context-dependent -- but it might look something like this in GitLab

pages:
  # fetch the whole repo
  # this logic can change if you're on a detached head, like an MR
  # so we account for that here
  before_script: | 
        if [[ -n "$CI_COMMIT_BRANCH" ]]; then  # branch pipelines
            git remote set-url origin "https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
            git fetch origin
            git checkout $CI_COMMIT_BRANCH
        fi
        if [[ -n "$CI_MERGE_REQUEST_IID" ]]; then  # MR pipelines
            git remote set-url origin "https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_MERGE_REQUEST_SOURCE_PROJECT_PATH}.git"
            git fetch origin
        fi
  script: |
        mkdir public
        branches=()
        # ref: https://stackoverflow.com/a/3847586/5747944
        eval "$(git for-each-ref --shell --format='branches+=(%(refname))' refs/heads/)"
        for branch in "${branches[@]}"; do
            git checkout "$branch"
            # build each branch and output to public directory
            # YOU implement this
            make build "$branch" -o "public/${branch}"
        done
  artifacts:
    paths:
      - public
  environment: # ensure outdated jobs are skipped
    name: pages # https://docs.gitlab.com/ee/ci/environments/deployment_safety.html#skip-outdated-deployment-jobs