If the application has been installed via the package manager, all you have to do is run

sudo apt-get remove <application_name>

That should always work. If the terminal isn't what stirs your tea, you could open System → Administration → Synaptic Package Manager, search for the package you want to remove, click on the checkbox next to it and select "mark for removal". Once you click "Apply", the package should be removed. There's of course also the Ubuntu Software Center. It's pretty much the same thing as Synaptic. Just search for the application name and click the "Remove" button.

Sometimes applications can be split up into multiple packages (for example, many games have a separate package for their music). To make sure that you uninstall all related packages AND configuration files, you can type

sudo apt-get purge <package-name>

or -in Synaptic- "mark for complete removal" instead of just "mark for removal".

As for applications that have been manually compiled and installed, there's not always a single way to remove them. The best thing to do is consult the README/INSTALL file that accompanied the source package - if one exists.

>I also want to remove any leftovers from my system after uninstall btw.

Then you'll have to manually clean all leftovers from your home.

Just run apt remove and don't care about the rest.

Use

sudo apt-get remove nameOfProgram

Or if you want to uninstall the program and also associated data:

sudo apt-get purge nameOfProgram

That will remove the program. It doesn't always remove the dependencies that were installed to make the program work. To clean up those package you can run this after:

sudo apt-get autoremove

Depending on your distro:

1. Using Fedora/CentOS/RHEL

   $ yum remove <package name>
   -or-
   $ rpm -e <package name>
   

2. Debian/Ubuntu/Mint/...

   $ apt-get --purge remove <package>
   -or-
   $ aptitude purge <package>
   -or-
   $ dpkg -r <package>
   

3. SuSE

   $ rpm -e <package>
   -or-
   $ zypper remove <package>
   -or-
   $ zypper rm <package>
   

4. BSDs

   $ pkg_delete <package>
   

5. Gentoo

   - uninstall a package, but only if it isn't required anymore by others -
   $ emerge -c <package>
   - uninstall a package, no matter whether it is still required by others -
   $ emerge -C <package>
   

6. ArchLinux

   $ pacman -Rc <package>
   

7. NixOS

   $ nix-env -e <package>
   

References

• zypper cheatsheet
• Debian: How to Install or Remove DEB Packages Using dpkg
• Chapter 5. Installing Applications: Packages and Ports
• Pacman Rosetta

Agree with other answers, and I wanted to clarify a bit. If my understanding is correct, typically ./configure is a script that makes sure / sets up your system so that compilation will go correctly. Then make runs a Makefile that actually compiles. Then make install runs the makefile with install as a parameter that actually copies your binaries and config files to the appropriate (as determined by the script author) system directories, which is why often make install must be run with elevated privileges.

Often the make script takes an uninstall parameter that actually erases everything that was copied to system directories. In my experience, this isn't always going to be a clean process. There's no fireproof way to roll back without ensuring yourself that all changes are tracked perfectly and writing the rollback script yourself.

In short, try make uninstall and if that doesn't work and you can't figure it out yourself, try posting on a mailing list or forum pertaining to the particular product in question.

Edit for more info: just running make should get you everything you need to run a program, as long as you keep your working directory as wherever you ran make from. That is, make will create all your binaries and config files, etc, and you can use the software fine from that directory. You won't have any globally accessible binaries or proper environment variables, though, if you don't copy things to system directories, such as with make install. So if you're just trying to run a self-contained binary that isn't software that something else will rely on, you don't actually need to run make install and won't have to worry about rolling back. Everything will be contained within your original working directory.