optimization - Speeding Up Python - Stack Overflow
How to optimize python code?
Python code optimization
Optimization Techniques in Python - Stack Overflow
Videos
Regarding "Secondly: When writing a program from scratch in python, what are some good ways to greatly improve performance?"
Remember the Jackson rules of optimization:
- Rule 1: Don't do it.
- Rule 2 (for experts only): Don't do it yet.
And the Knuth rule:
- "Premature optimization is the root of all evil."
The more useful rules are in the General Rules for Optimization.
Don't optimize as you go. First get it right. Then get it fast. Optimizing a wrong program is still wrong.
Remember the 80/20 rule.
Always run "before" and "after" benchmarks. Otherwise, you won't know if you've found the 80%.
Use the right algorithms and data structures. This rule should be first. Nothing matters as much as algorithm and data structure.
Bottom Line
You can't prevent or avoid the "optimize this program" effort. It's part of the job. You have to plan for it and do it carefully, just like the design, code and test activities.
Rather than just punting to C, I'd suggest:
Make your code count. Do more with fewer executions of lines:
- Change the algorithm to a faster one. It doesn't need to be fancy to be faster in many cases.
- Use python primitives that happens to be written in C. Some things will force an interpreter dispatch where some wont. The latter is preferable
- Beware of code that first constructs a big data structure followed by its consumation. Think the difference between range and xrange. In general it is often worth thinking about memory usage of the program. Using generators can sometimes bring O(n) memory use down to O(1).
- Python is generally non-optimizing. Hoist invariant code out of loops, eliminate common subexpressions where possible in tight loops.
- If something is expensive, then precompute or memoize it. Regular expressions can be compiled for instance.
- Need to crunch numbers? You might want to check
numpyout. - Many python programs are slow because they are bound by disk I/O or database access. Make sure you have something worthwhile to do while you wait on the data to arrive rather than just blocking. A weapon could be something like the
Twistedframework. - Note that many crucial data-processing libraries have C-versions, be it XML, JSON or whatnot. They are often considerably faster than the Python interpreter.
If all of the above fails for profiled and measured code, then begin thinking about the C-rewrite path.
Hi Pythonistas,
I'm interested in learning what optimization techniques you know for python code. I know its a general statement, but I'm interested in really pushing execution to the maximum.
I use the following -
-
I declare
__slots__in custom classes -
I use typing blocks for typing imports
-
I use builtins when possible
-
I try to reduce function calls
-
I use set lookups wherever possible
-
I prefer iteration to recursion
Edit: I am using a profiler, and benchmarks. I'm working on a library - an ASGI Api framework. The code is async. Its not darascience. Its neither compatible with pypy, nor with numba..
What else?
As I said in comment, you must start by finding what part of your code is slow.
Nobody can help you without this information.
You can profile your code with the Python profilers then go back to us with the result.
If it's a Web app, the first suspect is generally the database. If it's a calculus intensive GUI app, then look first at the calculations algo first.
But remember that perf issues car be highly unintuitive and therefor, an objective assessment is the only way to go.
ok, not entirely to the point, but before you go and start fixing it, make sure everyone understands the situation. it seems to me that they're putting some pressure on you to fix the "problem".
well first of all, when you wrote the application, have they specified the performance requirements? did they tell you that they need operation X to take less than Y secs to complete? Did they specify how many concurrent users must be supported without penalty to the performance? If not, then tell them to back off and that it is iteration (phase, stage, whatever) one of the deployment, and the main goal was the functionality and testing. phase two is performance improvements. let them (with your help obviously) come up with some non functional requirements for the performance of your system.
by doing all this, a) you'll remove the pressure applied by the finance team (and i know they can be a real pain in the bum) b) both you and your clients will have a clear idea of what you mean by "performance" c) you'll have a base that you can measure your progress and most importantly d) you'll have some agreed time to implement/fix the performance issues.
PS. that aside, look at the indexing... :)