Why we dropped Docker for Python environments
The best way to use docker with Python for development environments
is learning docker worth it for hobby Python development?
Any docker image with python and pip preinstalled?
TL;DR Docker is a great tool for managing software environments, but we found that it’s just too slow, especially for exploratory data workflows where users change their Python environments frequently.
We find that clusters depending on docker images often take 5+ minutes to launch. Ouch. In Coiled you can use a new system for creating software environments on the fly using only mamba instead. We’re seeing start times 3x faster, or about 1–2 minutes.
This article goes into the challenges we (Coiled) faced, the solution we chose, and the performance impacts of that choice.
https://medium.com/coiled-hq/just-in-time-python-environments-ade108ec67b6
Hello everyone,
I'm currently working on a project that involves multiple services: Django for the backend, PostgreSQL, and a front-end using Next.js. Maybe Redis and Celery in the future. Each of these services has its own GitHub repository and I've unified them using Git submodules.
Deployment using Dockerw (with Nginx and Gunicorn) works well. However, the problem is when it comes to setting up Docker for our development environment.
We're a team of three developers, and "ideally", I want each developer to be able to clone the repository and run docker-compose up to have all the services up and running. Also, I'd like the frontend developer to have the frontend directory binded for instant updates and the same for the backend developer and others.
The problem is with VsCode, especially with managing node_modules and Python packages within the Docker containers. VSCode doesn't recognize which packages are installed in the running containers (which is normal because it searches for the node_module in the local directory and it uses the host system Python interpreter).
For Next.js and React, I use a workaround by installing the packages in the container and then copying them to the local working directory using Dockerfile and Docker-compose configurations. However, with Python, I can't implement the same approach.
Thank you!
I've got a low frequency (6 pages per min limit) scraping script running on my pc more or less 24/7 to send me some push notifications. Now that I got the process proof of concept working with duct tape and sheer will, I've been slowly rebuilding it in parallel with better practices ex: it's just a loop with functions, no classes, testing, error catching or logging, I'm using selenium for everything but I'm moving to a mixture of selenium to load and click but faster selectorlib and easily updatelable yaml files to read the fully loaded HTML from selenium, etc, etc.
I dusted off my raspberry pi 4 and thought I could probably run a MySQL server for the data instead of a local SQLite database and maybe learn fastapi at the same time. In the past I just used the firewall to restrict to lan and had sqlalchemy connect directly to MySQL but fastapi seems like its worth learning and this seems like a good project to use it on.
I don't know what got me down the docker path but I've been researching it for about 2 weeks now to see if running a fastapi + MySQL dockers would make sense. after 2 weeks I can run containers and know the very basics of the dockerfile I still don't really understand what a development workflow in docker and vscode would look like as a single hooby dev. I'm starting to think maybe I should just run fastapi just with venv on uvicorn and MySQL without docker on the pi. Since I'm only accessing the server on lan, I'm not deploying on multiple devices or working as a team it's hard to see if knowing this extra layer is worth it. I might move the script itself over to the pi, I might make a simple crud web interface as well so maybe at that point docker will be helpful or will venv always suffice?