The following web link contains steps to increase the maximum heap for your AppMon Server, but since you need this for your Analysis Server, you would apply the same switch in the same manner, but to the config file called dtanalysisserver.ini instead of dtserver.ini:
In summary, here is what you need to do:
A switch must be added to override the Hard Coded limit.
The -Xms24G & -Xmx24G need the new switch -maxXmx=
You would add: -maxXmx=24G
For 24G, your dtanalysisserver.ini would have this:
And it does need the equal sign for that switch. And the min doesn't need to be 24G, it is set this way for illustration.
In release 5.5, this switch is already included in the ini file and it is preset to 0 so that there is no limit