The start script <install-path>/android-studio/bin/studio.sh does consult the files as follows to read configurations.

For 64bit systems:

<install-path>/android-studio/bin/studio64.vmoptions
${HOME}/.AndroidStudio<version>/config/studio64.vmoptions
<install-path>/android-studio.vmoptions

For 32bit systems:

<install-path>/android-studio/bin/studio.vmoptions
${HOME}/.AndroidStudio<version>/config/studio.vmoptions
<install-path>/android-studio.vmoptions

Depending on your architecture 32bit vs 64bit and how you did change the heap size setting, you can look through the files mentioned above and alter the Xmx with a text editor of your choice.

Thank you so much for myself. I got it in 2 hours. After searching too much on google, found nothing that works for me.

This is the only way to solve it.

Note : I'm using Ubuntu with 4g ram 64bit

Step 1. Go to file manager > show hidden files

Step 2. Go to folder > .config/Google/AndroidStudio-versionName

Step 3. Delete studio64.vmoptions file

Step 4. Restart Android Studio

-------EDIT--------

Android Studio 2.0 and above, you can create/edit this file by accessing "Edit Custom VM Options" from the Help menu.

-------ORIGINAL ANSWER--------

Open file located at

/Applications/Android\ Studio.app/Contents/bin/studio.vmoptions

Change the content to

-Xms128m
-Xmx4096m
-XX:MaxPermSize=1024m
-XX:ReservedCodeCacheSize=200m
-XX:+UseCompressedOops

Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool. Your JVM will be started with Xms amount of memory and will be able to use a maximum of Xmx amount of memory.

Save the studio.vmoptions file and restart Android Studio.

Note:

If you changed the heap size for the IDE, you must restart Android Studio before the new memory settings are applied. (source)

This is a configuration option in your projects "build.gradle" file. Look for something like this:

dexOptions {
    javaMaxHeapSize "4g"
}

I ran into this issue too after I haphazardly edited my VM options from within Android Studio without really paying attention to what I was copy-pasting off StackOverflow. In my case the error was this:

Invalid maximum heap size: -Xmx4096m # <------ increase this to most of your RAM
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

My machine has 8GB of RAM, so this helpful error message is instructing me to set this to a higher value.

In your case, -XmxheapSize is clearly not a valid value.

On Ubuntu I had to edit the offending line in this file:

~/.AndroidStudio3.1/config/studio64.vmoptions

(folder and filename may be different for you depending on which version of Android Studio you have and whether you're using 64 or 32bit.)

Run ls -a in your home folder to see all hidden folders, if you're not sure what your Android Studio config folder is named. See this for more details.

This method worked for me!

METHOD 2 :

Step 1 : Start Android studio and close any open project (File > Close Project).

Step 2 : On Welcome window, Go to Configure > Settings.

Step 3 : Go to Build, Execution, Deployment > Compiler

Step 4 : Change Build process heap size (Mbytes) to 512 and Additional build process to VM Options to -Xmx512m. Or any memory allocation in correspondent to your PC

Step 5 : Revalidate/Restart Android Studio. [![enter image description here][1]][1]

source: https://stackoverflow.com/a/39547461/4450923

I use 32 bit java installation

32 bit applications can't use 4g of RAM. Hence size exceeds the maximum representable size

You either need to use a smaller heap, or use 64 bit Java

Look my ScreenShot and add this line :

On build.gradle() add

android {
...
 dexOptions {
        javaMaxHeapSize "3g"

    }
..
}

And on gradle.properties add

org.gradle.jvmargs=-Xmx5120M

Update 2022; DSL element 'dexOptions' is obsolete and should be removed. It will be removed in version 8.0 of the Android Gradle plugin.
Using it has no effect, and the AndroidGradle plugin optimizes dexing automatically.

I tested and with Gradle 7.0+, increasing below mentioned org.gradle.jvmargs option is all we need.

Original Answer

I had a similar issue on Android Studio 2.3.3. APK packaging would fail saying Java heap space when building from Android Studio and trying to process a 300MB .so library. In my case two things were needed to fix it. Your build.gradle should contain something like that:

android {
    dexOptions {
        javaMaxHeapSize "4G"
    }
}

At the root of your project you need to have a file named gradle.properties containing the following:

# That's needed for java to be able the create the APK with our 300MB native  library
# Basically we grant java a 4GB heap.
org.gradle.jvmargs=-Xmx4608m

Click on Help -> Edit Custom VM Options.... If you don't have any VM options created, it'll prompt you to create one, accept it. Add / modify -Xmx with amount and type of data you want, like -Xmx8g for 8gb of heap. You should restart Android Studio afterwards so it'll take effect.

Edit: For more information about studio options, take a look at here.

Edit 2: If you have a problem with jps, make sure you added up $PATH for java home. See more details on this answer.

My Android Studio was working fine. But executing Gradle from command line gave this same error 'could-not-reserve-enough-space-for-1572864kb-object-heap'

Looks like, it was because Gradle couldn't find java

Defined JAVA_HOME system variable, pointing to the same path Android studio was using. (get it from Studio > project structure > SDK Location > JDK Location)

Worked for me.