You can use Google Cloud Scheduler which is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations.
The first 3 jobs per month are free. Next are $0.10 per job/month.
Answer from medvedev1088 on Stack Overflowgoogle cloud platform - How to run a Python script every day in GCP? - Stack Overflow
How to Schedule Python Script to Run Every 15 Minutes?
Python script scheduling on google cloud - Stack Overflow
Steps in deploying a python script in google scheduler - Stack Overflow
Videos
You can use Google Cloud Scheduler which is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations.
The first 3 jobs per month are free. Next are $0.10 per job/month.
Yes, you will need App Engine and the Cron service to schedule those scripts. You have some more or less straight-forward options to run those scripts on GCP:
- Deploy each one as different functions, and do something as mentioned in here: Basically deploy a simple function in App Engine that sends an HTTP request to your(s) function(s).
- Deploy each one as different handlers in App Engine, and schedule each calls.
- Deploy each one as different services and then schedule the calls, bearing in mind that, if you want to schedule a call to a different service, you must specify to which one, in your
cron.yamlfile, using thetargetkeyword and the name of the service.
Cloud functions aren't really suited to batch jobs that may be longer running than 10 minutes. I'd suggest running your job using a Compute Engine VM and scheduling it with a combination of Cloud functions / Cloud scheduler.
Here's a rough outline:
- Set up a containerized Compute Engine VM.
- Create a Cloud function to start the VM on a pub-sub trigger.
import googleapiclient.discovery
def start_job(event, context):
"""Triggered from a message on a Cloud Pub/Sub topic.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
compute = googleapiclient.discovery.build('compute', 'v1')
compute.instances().insert(
project='project_id',
zone='us-east1-b',
body=vm_config).execute()
- Create a Cloud Scheduler to trigger the pub-sub according to your schedule.
This lets you avoid the cost of an always-on VM. See this blog post for more detail.
Could this work?
import schedule
import time
def run_daily():
do something
do something else
schedule.every().day.at("08:20:30").do(run_daily) # HH MM SS
while True:
schedule.run_pending()
time.sleep(1)
Hey all!
Might be that this is a noob question, but I only now come into contact with Google Cloud platform and it's not my comfort zone, I'm more just the data guy. However, getting this setup is absolutely essential. I need to find a way to get my maintenance scripts running in fixed intervals 24/7 without anyone manually running them.
I am not that unfamiliar with Unix-based operating system, so I got a virtual machine on Cloud Compute with Debian, thinking it should be as easy as doing the scheduling in Python and letting the script run forever. But it seems that after some time it just stops. I really don't need anything too fancy, just to have the regular execution of the script.
Any advice is appreciated! Thanks!
I think a Cloud Scheduler may a be a good first solution/approach. It can, for example, make some http request, or push a message into a pubsub (which can be used as a pull or push trigger for your script).
Under the script I understand any required functionality. It can be implemented in many different ways - Cloud Function (or a group of different Cloud Functions working together to archive one goal), a Cloud Run, or anything else.
My usual personal preference is a pattern Cloud Scheduler => PubSub Topic => push Cloud Function. Other people may prefer other variations.
The choice of the solution (including the "script" implementation) in your case - I think - depends on functional and non functional requirements, context, scope, skills and knowledge of people who are to develop and maintain the solution, time, CAPEX and OPEX budget, etc.
Don't know if this is the best technically but I would go with a combination of Cloud Run and Cloud Scheduler (we currently have this combination running for one of our projects).
Cloud Run because your script seems to run just once a day and Cloud Run will basically go to sleep when it is not serving a request. This makes for lower overall cost i.e. it wakes up when it receives a request, executes the request and goes back to sleep (no charge to you when it is sleeping).
Cloud Scheduler to trigger the url endpoint on Cloud Run at 00:00am. As the name implies - Scheduler - schedules jobs to run at specific times.
I would also suggest securing your url endpoint (the one that will be deployed on Cloud Run). This ensures only your Cloud Scheduler is the one triggering the url (someone can not 'mistakenly' access the url over the internet and trigger the job unless they have the necessary privilege). We have a blog article about how to do this.
» pip install google-cloud-scheduler
I've been using Termux on my phone to do this so far, but I got tired of manually running the script (`termux-job-scheduler` doesn't work quite right, especially without a wakelock) or forgetting to and missing data.
I have a script that fetches data from a JSON endpoint and identifies entries that are new since last time it ran. I want to be notified (on my phone specifically) of those new entries when they occur (ideally, it should run every 30 minutes or more frequently).
I am looking for the cheapest solution to this that I can find since this is just a personal project (not even the kind I can use on a resume) and I don't really want to be paying significant fees for it. Open to non-GCP solutions as well, but worth noting I don't have a spare machine I can use as a server at home.