If I understand you correctly you have three columns with X and Y denoting the position of a value Z.
Consider the following example. There are three columns: X and Y contain positional information (categories in this case) and Z contains the values for shading the heatmap.
x = np.array(['a','b','c','a','b','c','a','b','c'])
y = np.array(['a','a','a','b','b','b','c','c','c'])
z = np.array([0.3,-0.3,1,0.5,-0.25,-1,0.25,-0.23,0.25])
Then we create a dataframe from these columns and transpose them (so x,y and z actually become columns). Give column names and make sure Z_value is a number.
df = pd.DataFrame.from_dict(np.array([x,y,z]).T)
df.columns = ['X_value','Y_value','Z_value']
df['Z_value'] = pd.to_numeric(df['Z_value'])
resulting in this dataframe.
X_value Y_value Z_value
0 a a 0.30
1 b a -0.30
2 c a 1.00
3 a b 0.50
4 b b -0.25
5 c b -1.00
6 a c 0.25
7 b c -0.23
8 c c 0.25
From this you cannot create a heatmap, however by calling df.pivot('Y_value','X_value','Z_value') you pivot the dataframe to a form that can be used for a heatmap.
pivotted= df.pivot('Y_value','X_value','Z_value')
The resulting dataframe looks like this.
X_value a b c
Y_value
a 0.30 -0.30 1.00
b 0.50 -0.25 -1.00
c 0.25 -0.23 0.25
You can then feed pivotted to the sns.heatmap to create your heatmap.
sns.heatmap(pivotted,cmap='RdBu')
Resulting in this heatmap.

You may need to make some adjustments to the code for your precise needs. But since I had no example data to go from I needed to make my own example.
Answer from error on Stack OverflowAdditional Example: Adding annotations to highlight specific regions
Additional Example: Handling Irregularly Spaced Data
Look like you use pandas dataframe. Before plotting pivot dataframe to be a table and use heatmap method, i.e. from seaborn:
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_clipboard()
table = df.pivot('Y', 'X', 'Value')
ax = sns.heatmap(table)
ax.invert_yaxis()
print(table)
plt.show()

Output:
X 0 1
Y
1 0.6 0.8
2 0.3 0.4
3 0.2 0.9
I'm late to the party, but here's a solution to OP's question using only matplotlib, as was requested. It relies on using imshow's extent option to do the job. For more info, see the matplotlib manual.
import matplotlib.pyplot as plt
xvals = [0,1]
yvals = [1,3]
zvals = [[0.6, 0.3, 0.2], [0.8, 0.4, 0.9]]
heatmap, ax = plt.subplots()
im = ax.imshow(zvals,cmap='inferno',extent=[xvals[0],xvals[1],yvals[0],yvals[1]],interpolation='nearest',origin='lower',aspect='auto')
ax.set(xlabel='some x', ylabel='some y')
cbar = heatmap.colorbar(im)
cbar.ax.set_ylabel('stuff')
heatmap.savefig('heatmap.png')
which results in the following image

My answer is a small edit to answer provided by warped.
The difference is the inclusion of the parameter 'extend' in the contourf method call.
https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.contourf.html
For some more information about the minimum/maximum color-map behaviour, see:
https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/contourf_demo.html#sphx-glr-gallery-images-contours-and-fields-contourf-demo-py
# libraries
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import kde
from scipy.interpolate import griddata
# Get the data (csv file is hosted on the web)
url = 'https://raw.githubusercontent.com/ampil/sandbox/master/latest-sales-sample.csv'
df = pd.read_csv(url, sep = ';')
df = df.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)
# create data
x = df['X']
y = np.log(df['Y'])
z = df['cluster']
#following 2 lines provided by user-warped
xi = np.arange(0, np.max(x), 0.1)
yi = np.arange(0, np.max(y), 0.1)
xi, yi = np.meshgrid(xi,yi)
# interpolate
zi = griddata((x,y),z,(xi,yi),method='cubic')
#define color map...which you can choose to modify with 'set_under' and 'set_over'
#as per: https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/contourf_demo.html#sphx-glr-gallery-images-contours-and-fields-contourf-demo-py
cmap = plt.cm.get_cmap("coolwarm")
fig = plt.figure()
ax = fig.add_subplot(111)
ax.axis((x.min(), x.max(), y.min(), y.max()))
#added the 'extend' parameter to user:warped edit as per documentation of plt.contourf
#https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.contourf.html
plt.contourf(xi, yi, zi, levels=[1,2,3,4,5,6], cmap=cmap, extend='both')
plt.plot(x, y,'k.')
plt.xlabel('x',fontsize=16)
plt.ylabel('y',fontsize=16)
plt.show()

As far as extending the colours beyond what they are now...you'd get a result that would look very different from the main area and would likely have little meaning. If this were a GIS application, I would have those exterior pixels to be "NODATA".
EDIT: Providing evidence that filling the exterior would look strange...
Using a canned gdal method to fill nodatas, this is what it would look like:

This was quick and dirty and other methods likely exist but would probably look equally odd. Perhaps numpy.nan_to_num is another solution if you don't have gdal.
In case you're curious...here's the code (continues from previous code block):
import gdal
ds = gdal.GetDriverByName('MEM').Create('', zi.shape[1], zi.shape[0], 1, gdal.GDT_Float32)
in_band = ds.GetRasterBand(1)
in_band.SetNoDataValue(-9999)
in_band.FlushCache()
raster_data = np.copy(zi)
raster_data[np.isnan(zi)] = -9999
in_band.WriteArray(raster_data)
#this line takes a while to run...grab a coffee
result = gdal.FillNodata(in_band, None, maxSearchDist=20000, smoothingIterations=0)
in_band.FlushCache()
newz = in_band.ReadAsArray()
fig = plt.figure()
ax = fig.add_subplot(111)
ax.axis((x.min(), x.max(), y.min(), y.max()))
#added the 'extend' parameter as per documentation of plt.contourf
#https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.contourf.html
plt.contourf(xi, yi, newz, levels=[1,2,3,4,5,6], cmap=cmap, extend='both')
plt.plot(x, y,'k.')
plt.xlabel('x',fontsize=16)
plt.ylabel('y',fontsize=16)
plt.show()
import matplotlib
colors = ['red','green','blue','purple','black', 'coral']
fig = plt.figure(figsize=(8,8))
plt.scatter(x, y, c = z, cmap=matplotlib.colors.ListedColormap(colors))
plt.show()

OK, there's a few steps to this.
First, a much simpler way to read your data file is with numpy.genfromtxt. You can set the delimiter to be a comma with the delimiter argument.
Next, we want to make a 2D mesh of x and y, so we need to just store the unique values from those to arrays to feed to numpy.meshgrid.
Finally, we can use the length of those two arrays to reshape our z array.
(NOTE: This method assumes you have a regular grid, with an x, y and z for every point on the grid).
For example:
import matplotlib.pyplot as plt
import numpy as np
data = np.genfromtxt('eye_.txt',delimiter=',')
x=data[:,0]
y=data[:,1]
z=data[:,2]
## Equivalently, we could do that all in one line with:
# x,y,z = np.genfromtxt('eye_.txt', delimiter=',', usecols=(0,1,2))
x=np.unique(x)
y=np.unique(y)
X,Y = np.meshgrid(x,y)
Z=z.reshape(len(y),len(x))
plt.pcolormesh(X,Y,Z)
plt.show()
In case you don't have a regular grid (i.e. a value of z for each meshgrid value of x and y), you can use a more general method based on pandas data frames:
import pandas as pd
import seaborn as sns
import matplotlib.pypot as plt
data = pd.DataFrame(data={'x':x, 'y':y, 'z':z})
data = data.pivot(index='x', columns='y', values='z')
sns.heatmap(data)
plt.show()
The pivot method will use unique values from index and columns to construct a table with missing measurments set to NaN. The table can then be plotted as a heatmap.
Use pivot and seaborn.heatmap:
import seaborn as sns
sns.heatmap(df.pivot(index='Y', columns='X', values='Z'))
Output:

IF you want to handle missing coordinates:
df2 = (df
.pivot(index='Y', columns='X', values='Z')
.pipe(lambda d: d.reindex(index=range(d.index.min(), d.index.max()+1),
columns=range(d.columns.min(), d.columns.max()+1),
)
)
)
sns.heatmap(df2)
Output:

https://seaborn.pydata.org/generated/seaborn.heatmap.html
However I notice your data are all numeric. You may be looking for a Z dimensional colored scatterplot rather than a true 'heatmap'. You can use the plt.scatter function of matplotlib. (X_col=x,Y_col=y,color=Z). https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html


