After a lot of research, I found an easy way of eliminating and manipulating specific parts of the predefined controls.

Create your elements as you usually would, like so:

<audio autoPlay>
    <source src='audioUrl' type='audio/mpeg' />
</audio>

Then in the CSS file, you write the following:

/* Specifies the size of the audio container */
audio {
  width: 115px;
  height: 25px;
}

audio::-webkit-media-controls-panel {
  -webkit-justify-content: center;
  height: 25px;
}

/* Removes the timeline */
audio::-webkit-media-controls-timeline {
  display: none !important;
}

/* Removes the time stamp */
audio::-webkit-media-controls-current-time-display {
  display: none;
}
audio::-webkit-media-controls-time-remaining-display {
  display: none;
}

With this code, you should get a small and nice looking container with only mute-button, pause/play-button and the 'download-file'-tag.

For an overview of all the things you can modify, have a look here.

The following code will also remove the mute- and the play-button:

/* Removes mute-button */
audio::-webkit-media-controls-mute-button {
  display: none;
}

/* Removes play-button */
audio::-webkit-media-controls-play-button {
  display: none;
}