The 560 is pretty anemic, compared to today’s cards, but it is absolutely better than the 5770, which will not have any acceleration under Mojave. Hopefully you got the 4GB card, but the 2GB is still way better than the disappointing 5770.
You do no explicitly state, but imply that the 560 is a PC card and thus you can just pop it in and, apart from not seeing the normal startup screen, use it as is. But, as discussed, if you retain the 5770, the Mac will simply force the boot screen to the 5770, regardless of what slot it is in; once you are booted into your Desktop, you should go into Displays prefs and designate the display on the 560 as the Main Screen by dragging the Menubar from the 5770 display to the 560 display.
After that, you will see your Startup Screen and progress bar on the 5770 display (boot/startup display), until just before Login Screen (if enabled) or Desktop, when it will “leap” to your 560 display (Main Screen). Not elegant, but fully functional.
And, again, the 5770 isn’t strictly required at all; you can opt to run both (or one or three or whatever) displays strictly from the 560, so that all displays have full acceleration and resolution options. If you don’t game or do FCP and the like or care about frame rates or resolutions or HiDPi or refresh rates on the second display, you can keep the 5770 (or other OEM Apple card) for the comfort of seeing a Startup Screen.
If you do decide you would be uncomfortable with a blank Startup Screen, whether solo or not, the 560/570/580 can be flashed at no cost using a Windows PC. Many places will do it for $150 and up; the instructions are bone simple and you can do it yourself in under ten minutes with a pal’s Windows box. For anyone reading, I will happily do it for free if you pay shipping.
Finally, just to reiterate, I recommend placing the faster 560 in Slot 1, and the 5770 in Slot 2, unless you need that 16x slot for something like a RAID or other drive accelerator (I have a RocketRaid RR840 hosting 12 of 16 available SATA channels with fast SSDs striped into a wicked fast RAID; I use a pathetic OEM GT120 in Slot 4 for a boot screen and host for two additional 1440p displays); if Slot 2 is occupied, the 5770 will have to go in Slot 3, and it will block Slot 4 due to its width.
You will find two 6-pin power ports on the motherboard side by side – one already occupied by the 5770 – use the second for your 560. If your 560 has an 8-pin power input, you will need a ~$6 6-pin to 8-pin power cable (https://www.amazon.com/gp/product/B00OSLGQGE/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1). Do not try to just use a 6-to-6 cable; it will fit and it will boot, but when you try to push the card hard, you will get kernel panics.
I don’t think you mentioned what kind of work/apps you employ; if you are aching for power, I would still consider an nVidia card and sticking with High Sierra for more performance and GPU Compute for certain apps; but if you need Metal, not CUDA, go AMD.