First off, javascript in the client is probably not the best language to do this nor the best approach to make this happen. It might work, but it's better to know what is best when choosing an approach to a problem. Also, it will avoid for you clicking ~800 times in the popup accepting the download.

You can get the files in a programatically way by just learning what you browser is doing to get the file and trying to reproduce it in bunch.

After inspecting the calls you can see that it's calling an endpoint and that endpoint is returning a link which contains the file that you can download.

Well, that is going to be easy, so now you just need to make the script in any language to be able to retrieve them.

I've chosen javascript but not client side, but nodejs which means that this has to run from your computer.

You could do the same with bash, python or any other language.

To run this do the following:

• Go to a new empty directory
• Run npm install axios
• Create a file with the code I pasted let's call it crawler.js
• Run node crawler.js

This has been tested using node v8.15.0

// NOTE: Require this to make a request and save the link as file 20190813:Alevale
const axios = require('axios');
const fs = require('fs');

let now = new Date();
let daysOfYear = [];
const baseUrl = 'https://a4dzytphl9.execute-api.ap-southeast-1.amazonaws.com/prod/eod/'

for (var d = new Date(2016, 0, 1); d <= now; d.setDate(d.getDate() + 1)) {
    daysOfYear.push(new Date(d).toISOString().substring(0,10));
}

const waitFor = (time) => {
    return new Promise((resolve => setTimeout(resolve, time)))
}

const getUrls = async () =>{
    let day
    for (day of daysOfYear) {
        console.log('getting day', baseUrl + day)
        // NOTE: Throttle the calls to not overload the server 20190813:Alevale
        await waitFor(4000)

        await axios.get(baseUrl + day)
            .then(response => {
                console.log(response.data);
                console.log(response);
                if (response.data && response.data.download_url) {
                    return response.data.download_url
                }
                return Promise.reject('Could not retrieve response.data.download_url')
            })
            .then((url) =>{
                axios({
                    method: 'get',
                    url,
                    responseType: 'stream'
                })
                    .then(function (response) {
                        // NOTE: Save the file as 2019-08-13 20190813:Alevale
                        response.data.pipe(fs.createWriteStream(`${day}.csv`))
                    })
                    .catch(console.error)

            })
            .catch(error => {
                console.log(error);
            });
    }
}

getUrls()

Download File from URL

There are a couple ways to do this. As mentioned, using the developer tools could work (more likely it will give you the url to the file) and right-clicking the link will work. Alternatively there are these options.

In Chrome

1. Go to the URL
2. Right-click the webpage
3. Select Save As...

For verification purposes, here are png, jpg, and mp3 links. Follow them and try these steps. However, in my experience. If you already have a url to a file, opening up Chrome and following these steps is rather tedious so here is an alternative.

In Command Line

1. Open your favorite terminal emulator
2. type curl -O URL

• Where O is written in capital
• And URL is the URL to the file, e.g. http://example.com/file.mp3

What you could do - setup a certain parameter for example: https://example.com/?autodownload=1

then add a piece of javascript that either imediately, or after some timeout automatically downloads the file. Something like:

document.addEventListener("DOMContentLoaded", function(event) { 
    if (window.location.href.indexOf("autodownload") > -1)
      setTimeout(function(){ 
            window.open("https://example.com/docs/some_file.xlsx");
        }, 3000); //your timeout in miliseconds
    }
});

of course this solution will not work, if browser will block opening new windows. Second possibility with this approach is to replace the window.open function with click on the anchor itself.

Dear User,

Thank you for reaching out Microsoft 365 community. To get the file "download" instead of "open", please modify the download URL like below.

https://.sharepoint.com/sites//_layouts/download.aspx?SourceUrl=

Eg.

https://contoso.sharepoint.com/sites/siteA/_layouts/download.aspx?SourceUrl=https://contoso.sharepoint.com/sites/siteA/Shared%20Documents/sample.xlsx

For the , please locate the file on sharepoint > right click it > Details > scroll to bottom and copy path.

We are committed to ensuring you have a smooth experience with Microsoft 365 products, and we appreciate your patience and understanding as we work through this challenge.

If there is anything further we can assist you with in the meantime, please do not hesitate to reach out.

Best regards,

Sophia

Evidently I was wrong. pythonanywhere is really easy. (But you must use any paid plan in order to access most outside servers (i.e. get files from any website you want)). As of right now, the cheapest plan they have is $5 a month.

Stackoverflow isn't a code-writing service. But enjoy this little exception.

here's a basic implementation of what you've asked for:

We're gonna use an HTTP server written in Python, using a framework called Flask. That server will provide two functions:

1. Serve us that basic form you've described
2. allow us to request from the backend the file we want, and it will force our web browser to download it (and not display it, for example)

Step 1

Create a "pythonanywhere" account. Attention: your website will be hosted on username.pythonanywhere.com, choose wisely.

hit "open web tab"*

Step 2

Select the latest Flask version in "add new web app" (3.10 in this case) and hit Next.

Step 3

verify the website works, click on that link, you should see "Hello from Flask!".

Step 4

Go to the "Files" tab, enter the mysite directory

and create a new directory called "templates"

Step 5

Create a new file called index.html, and put this html in it:

(css and form taken from w3schools)

<!DOCTYPE html>
<html>
<style>
input[type=text], select {
  width: 100%;
  padding: 12px 20px;
  margin: 8px 0;
  display: inline-block;
  border: 1px solid #ccc;
  border-radius: 4px;
  box-sizing: border-box;
}

input[type=submit] {
  width: 100%;
  background-color: #4CAF50;
  color: white;
  padding: 14px 20px;
  margin: 8px 0;
  border: none;
  border-radius: 4px;
  cursor: pointer;
}

input[type=submit]:hover {
  background-color: #45a049;
}

div {
  border-radius: 5px;
  background-color: #f2f2f2;
  padding: 20px;
}
</style>
<body>

<h3>File Proxy</h3>

<div>
  <form method="POST" action="/download"> <!-- notice the @app.route("/download"...) in the flask file -->
    <label for="fpath">File Path</label>
    <input type="text" id="fpath" name="fpath" placeholder="Path...">  
  
    <input type="submit" value="Submit">
  </form>
</div>

</body>
</html>

Hit save

Step 6

Go back to the "Files" tab, and enter mysite directory again.

edit the flask_app.py file

and replace the code in it with the following:

from flask import Flask, request, render_template, Response
import requests
app = Flask(__name__)

content_type_to_extension={"text/html": ".html"}

FALLBACK_EXTENSION=".bin"

# get the path from the form, GET it ourselves, and return it with a content-type that will force the client's browser to download it and not attempt to display it.
@app.route("/download", methods=["POST"])
def loadData():
    data = request.form["fpath"]
    
    r = requests.get(data)
    response = Response(
        r.content,
        status=r.status_code,
        content_type="application/stream"
        # content_type=r.headers['content-type']
    )
 
    # basic naming attempt. if it has a uri (text after the last /), let's (maybe falsly, this is a POC) assume that it has a file extension. (you can add regex tests and whatnot)
    try:
        fname = data.replace('://','')
        fname = fname[fname.rindex("/")+1:]
        assert (len(fname) > 1)
    except (IndexError, ValueError, AssertionError):
        fname = data.replace("/", "-")
        ext = content_type_to_extension.get(r.headers['content-type'].split(";")[0], FALLBACK_EXTENSION) # if we can't find ther correct extension, fallback to .bin
        fname += ext

    response.headers["Content-Disposition"] = f'attachment; filename={fname}'
    return response


@app.route("/")
def index():
    return render_template("index.html") # that html file in /templates

if __name__ == "__main__":
    app.run(host='0.0.0.0')

Hit save

Step 7

Go back to the "web" tab, and hit that green "reload" button. (or the blue reload icon next to the save in the flask_app.py edit panel)

And go to your website.

The requests library requires a schema to be specified. so don't forget the http(s):// or add it via python.

If you don't have a paid account, checking out the error logs in the "web" tab tells us

OSError('Tunnel connection failed: 403 Forbidden')))
**NO MATCH**

or

Network is unreachable

So you gotta go with a paid plan. Or find some other method to host your web server.

For the record - what we're doing here is creating a "dumbed-down" web gui version of a cli http client, like wget or curl.

you could just run in bash / powershell wget <your-url> and it would be basically the same.