GDB uses debuginfod_find_debuginfo() to find and download the debug info files. Documentation says:

debuginfod_find_debuginfo(), debuginfod_find_executable(), and debuginfod_find_source()
query the debuginfod server URLs contained in $DEBUGINFOD_URLS
(see below) for the debuginfo ...
...

CACHE
If the query is successful, the debuginfod_find_*() functions save the target
file to a local cache. The location of the cache is controlled by the
$DEBUGINFOD_CACHE_PATH environment variable (see below).
Cleaning of the cache is controlled by the cache_clean_interval_s and
max_unused_age_s files, which are found in the $DEBUGINFOD_CACHE_PATH directory.
cache_clean_interval_s controls how frequently the cache is traversed for cleaning
and max_unused_age_s controls how long a file can go unused
(fstat(2) atime) before it's removed from the cache during cleaning.

So it seems like you are suffering from either too frequent cleaning, or too low max_unused_age_s setting.

And unsetting DEBUGINFOD_URLS in the environment should stop GDB from downloading anything.