Technically speaking, you could put your requirements.yml file anywhere you like as long as you reflect the correct path in your ansible-galaxy install command.

Meanwhile, if you ever want to run your playbooks from Ansible Tower/Awx, I suggest you stick to the Ansible Tower requirements and put your requirements.yml file in <project-top-level-directory>/roles/requirements.yml

Regarding dependencies between roles, ansible-galaxy is able to follow them by itself when they are encountered during installation. So you don't need to specify all of them in your requirements.yml, only top level ones. You just need to specify your dependencies correctly in each external roles.

In meta/main.yml for role1

dependencies:
  - src: https://my.scm.com/my-ansible-roles/role2.git
    scm: git
    version: master
    name: role2
  - src: https://my.scm.com/my-ansible-roles/role3.git
    scm: git
    version: master
    name: role3

In meta/main.yml for role2

dependencies:
  - src: https://my.scm.com/my-ansible-roles/role4.git
    scm: git
    version: master
    name: role4
  - src: https://my.scm.com/my-ansible-roles/role5.git
    scm: git
    version: master
    name: role5

roles/requirements.yml

---    
- src: https://my.scm.com/my-ansible-roles/role1.git
  scm: git
  version: master
  name: role1

To be as exhaustive as possible, this is what I now usually do on my projects to handle dependencies locally as well as local/project only roles

Basic project structure

ansible-project-dir
└─── roles
|    └─── locally-versioned-role1
|    └─── locally-versioned-role2
|    └─── ...
|    └─── requirements.yml
|    └─── .gitignore
└─── ansible.cfg
└─── playbook1.yml
└─── playbook2.yml

ansible.cfg

I force roles search and downloads in the local roles directory by setting roles_path = roles, so user can use ansible-galaxy install without the -p parameter.

roles/requirements.yml

Already discussed above. Just list dependencies to top-level external (i.e. not versioned in the project) as galaxy role name or as git uris. If you need to fully checkout those roles to later make git commits/push on them, you can use ansible-galaxy install -g -f -r roles/requirements

roles/.gitignore

# Ignore everything in roles dir
/*
# Except:
# the .gitignore file
!.gitignore
# the requirements file
!requirements.yml
# Readme if you have one
!README.md
# and any specific role we want to version locally
!locally-versioned-role*/

@SwiperNo, I feel your pain. Reading your question, I can see that you have the right aspects for using galaxy with AWX. In my experience, and long hours of testing / troubleshooting, the following worked for me:

1. A gitlab or github project with a "collections" folder and a "requirements.yml" within it.
2. Contents of the requirements.yml might be:

--- 
collections:
- name: chocolatey.chocolatey
  source: https://galaxy.ansible.com

3. Setup AWX with a "Source Control" credential using a token (Token generated from Gitlab/Github)

4. Setup an AWX Project with the "Source Control Type" as "Git" and use the "Source Control Credential" created in step 3. I also ticked the option "Update Revision on Launch" to update the Git source in the cache a runtime.

5. Ensure you have a PersistentVolume attached to the AWX instance. The Git source and collections need to cache it locally from what I understand and believe this could be part of your problem. See the following link re PersistentVolume, you can add the config to the awx.yml file, which you would have used to build the K3s instance at the begining. https://docs.ansible.com/automation-controller/4.0.0/html_zh/administration/operator_advanced_configurations.html#persist-projects-directory

6. Lastly, I would recommend enabling verbose debugging within the template and see if there are errors which pertain to caching or the requirements.yml.

For reference, AWX build using K3s. https://github.com/ansible/awx-operator#persisting-projects-directory