You would be best off by obtaining a schematic to aide in your repair efforts.
Anything that even looks like an eletrolytic cap should be replaced, as should any of the old wax/paper/oil caps that might be in there, just to be careful.
The 20/20/20 cap with one section having a 25V rating could be a 20/20/20 @ 525, or you could do a 20/20 at high voltage and just use a 20 @ 25V electrolyic on its own (they didn't back in the day because they were somewhat large and hard to stash under the chassis).
Is the 4uF/600V cap also a can? If you have a tube rectifier, this may be the first cap in the power supply, in which case going larger may cause some problems. There are lots of 4.7uF caps in various voltages, as well as some 3.9uF Solens that may work for you. The schematic would tell you for sure, if it's some kind of power supply decoupling cap, then you could go larger, and maybe even a lower votlage, it just depends.
I'd dump the square mica capacitors if you can find a nice film cap to replace them.
Good luck with the project, there is a lot of learning to be had by messing with these old amps!
-PB