multiprocessing.Pool can handle this. By default, the pool will create one process per CPU on your machine. Since they all will spend some time waiting on the disk to read the files, its a good way to mostly commit your machine. Its something you can tweak if either CPU usage or disk usage makes your machine unresponsive to other programs.

The pool divides work into chunks much like your example, but in the case of heavy processing on potentially variable sized files, a chunksize of 1 can avoid you wating for the tall-pole process to complete.

import multiprocessing as mp

def do_the_work(filename):
    pass

master_list = [file1, file2, fil3, ...filen]
with mp.Pool() as pool:
    results = pool.map(do_the_work, master_list, chunksize=1)

If you are restricted to windows you can write a DOS sript and run it in parallel as decribed parallel execution of shell processes

If can run in linux there is a procedure pretty well worked out using gnu-parallel. I did something similar for QGIS described in more detail at How to run processing commands in parallel in QGIS

Although the examples I give are for QGIS they would work in principle for arcpy in DOS too. ie If you can generate the the proces you neeed to carry out on one file as a BASH script which incorporates the file names as inputs, then you can use gnu parallel in linux or a parallel DOS script in DOS to handle the parallel work. Based on Oles answer to: parallelising-gis-operations-in-pyqgis eg:

Assuming you had a bash script which could run your "near" process as follows

my_standalone_script.py /path/to/a/point/shpfile.shp /path/to/another/point/file.shp /path/to/output/file.shp

You could install gnu parallel and run the script in parallel with:

find /path/to/point/shpfiles -type f -name *.shp | parallel my_standalone_script.py {} /path/to/another_point_file.shp /path/to-output/folder/neared_{/}.shp

As far as reasonable time is concerned gnu parallel will carry out the operation about as well as could be be achieved given any hardware limitations. You can even cluster more computers together and have them all carry out some of the solution each. (provided you can install the software required on each unit).

If your file names are in a list, you could use this code:

from multiprocessing import Process

def YourCode(filename, otherdata):
    # Do your stuff

if __name__ == '__main__':
    #Post process files in parallel
    ListOfFilenames = ['file1','file2', ..., 'file1000']
    ListOfProcesses = []
    Processors = 20 # n of processors you want to use
    #Divide the list of files in 'n of processors' Parts
    Parts = [ListOfFilenames[i:i + Processors] for i in xrange(0, len(ListOfFilenames), Processors)]

    for part in Parts:
        for f in part:
            p = multiprocessing.Process(target=YourCode, args=(f, otherdata))
            p.start()
            ListOfProcesses.append(p)
        for p in ListOfProcesses:
            p.join()

The python standard library offers two process pools to simplify what you're trying to achieve: multiprocessing.Pool (as mentioned by @RootTwo) and concurrent.futures.ProcessPoolExecutor.

I personally like the second one (despite its more limited API) because you can replace it with a ThreadPoolExecutor without changing the rest of the code. Threads are better suited if your processing is I/O bound, because inter-process communication is slow and processes are slower to start (especially on Windows). With the concurrent.futures API, you can easily swap the two and test which one works better.

There are some other ways you can improve the code, most notably:

1. If you're on Python 3.x as you should be, I suggest the use of pathlib instead of os.path.
2. Put code from the if __name__ == "__main__": into a "main" function to make automated testing easier.
3. Use snake_case instead of camelCase as that's the accepted standard for Python.
4. (Opinion) If you're on python 3.x, you could also use type annotations to improve the readability of the code.

With this in mind, I would rewrite your code as:

from pathlib import Path
from concurrent.futures import ProcessPoolExecutor, as_completed
from typing import Iterable

def get_files(directory: Path) -> Iterable[Path]:
    # Using glob simplifies the code in these cases.
    return (file for file in directory.glob("**/*") if file.is_file())

def foo(file: Path):
    ...

# This can be easily called from a test
def process_files(directory: Path):
    # I am using sum, so that the result is computed lazily and we
    # do not need to build a list of all files. If the directory is
    # very large, this could save a lot of memory.
    # Since get_files returns a one-shot generator, we cannot
    # save it to a variable and reuse it here and below.
    file_count = sum(1 for _ in get_files(directory))
    
    with concurrent.futures.ProcessPoolExecutor() as executor:
        futures = executor.map(foo, get_files(directory))
        # Reading the value returned by the executor is very
        # important, because exceptions happening in the `foo`
        # function will be captured and not be raised until the
        # value is read, thus obfuscating the errors.
        #        
        # Another nice solution for the progress would be
        # using the library tqdm.
        for i, _ in enumerate(as_completed(futures)):
            print(f"Processed file {i+1} / {file_count}")

if __name__ == "__main__":
    process_files(Path(""))

When considering performance of any program, you have to work out if the performance is bound by I/O (memory, disk, network, whatever) or Compute (core count, core speed, etc).

If I/O is the bottleneck, there's no point having multiple processes, a faster CPU, etc.

If the computation is taking up all the time, then it is worth investing in multiple processes, etc. "Computation time" is often diagnosed as being the problem, but on closer investigation turns out to be limited by the computer's memory bus speed, not the clock rate of the cores. In such circumstances adding multiple processes can make things worse...

Check

You can check yours by doing some performance profiling of your code (there's bound to be a whole load of profiling tools out there for Python).

My Guess

Most of the time these days it's I/O that's the bottleneck. If you don't want to profile your code, betting on a faster SSD is likely the best initial approach.

Unsolvable Computer Science Problem

The architectural features of modern CPUs (L1, L2, L3 cache, QPI, hyperthreads) are all symptoms of the underlying problem in computer design; cores are too quick for the I/O we can wrap around them.

For example, the time taken to transfer 1 byte from SDRAM to the core is exceedingly slow in comparison to the core speed. One just has to hope that the L3, L2 and L1 cache subsystems have correctly predicted the need for that 1 byte and have already fetched it ahead of time. If not, there's a big delay; that's where hyperthreading can help the overall performance of the computer's other processes (they can nip in and get some work done), but does absolutely nothing for the stalled program.

Data fetched from files or networks is very slow indeed.

File System Caching

In your case it sounds like you have 1 single input file; that will at least get cached in RAM by the OS (provided it's not too big).

You may be tempted to read it into memory yourself; I wouldn't bother. If it's large you would be allocating a large amount of memory to hold it, and if that's too big for the RAM in the machine the OS will swap some of that RAM out to the virtual memory page file anyway, and you're worse off than before. If it's small enough there's a good chance the OS will cache the whole thing for you anyway, saving you the bother.

Written files are also cached, up to a point. Ultimately there's nothing you can do if "total process time" is taken to mean that all the data is written to disk; you'd be having to wait for the disk to complete writing no matter what you did in memory and what the OS cached.

The OS's filesystem cache might give an initial impression that file writing has completed (the OS will get on with consolidating the data on the actual drive shortly), but successive runs of the same program will get blocked once that write cache is full.

If you do profile your code, be sure to run it for a long time (or repeatedly), to make sure that the measurements made by the profiler reveal the true underlying performance of the computer. If the results show that most of the time is in file.Read() or file.Write()...

To fully utilize your hardware core, it's better to use the multiprocessing library.

from multiprocessing import Pool

from os import listdir 
import csv

def process_file(file):
    #load the text file as list using csv module 
    #run a bunch of operations
    #regex the int from the filename. for ex file1.txt returns 1, and file42.txt returns 42
    #write out a corresponsding csv file in dirB. For example input file file99.txt is written as out99.csv

if __name__ == '__main__':
    mypath = "some/path/"

    inputDir = mypath + 'dirA/'
    outputDir = mypath + 'dirB/'

    p = Pool(12)
    p.map(process_file, listdir(inputDir))

Document of multiprocessing: https://docs.python.org/2/library/multiprocessing.html