I do not have a Mac to test but I believe it detects you are running this on a Mac and so grays out this option and presents you with the instructions to set it up on a Mac. On PC the button is active and I do not have the note about setting it up on a Mac.
Have you looked at the instructions? I think they will link you to the Linux agent with instructions on how to install it.
We were also facing the same issue and resolved it.
On MAC, local machine option is disabled and so cant be selected.
What I did is, I selected "On a remote machine" >> "Any OS(as MAC OS is not available)" >> Any JVM Version. Clicked on Next.
It gave me option to download the agent as per my selected OS and JVM version, but this agent is not required to be downloaded because when I downloaded the full dynatrace (dynatrace-full-126.96.36.1999-darwin-x86.jar) it automatically downloads the agent on your downloded path.
So I clicked on Next without downloading agent and selected the Arguments given for connecting agent and modified path according to the installation structure.
After restarting the server I was able to connect agent successfully.