We frequently had a problem where the build was constantly red, atleast one test had failed. From a Pm's point of view they stop seeing the benefit of automation, testers themselves start to lose hope that they will keep see a green build and worst of all the results stop being valued by the team.
We started by moving tests that were flaky or had defects attached to them into a separate run. We continued to execute these defect/flaky tests, looking to see whether the defect tests failed earlier or started to pass and making sure that the flaky tests were flaky.
Unfortunately quite often during projects testers are pressed for time, so with this steup we had tests still sitting in the regression suite waiting to be investigated whether the failure was the result of the flaky test or a real defect. This lead us to creating a final run called investigation, where all tests that had failed in the previous regression run are rerun. These are ususally run straight after completion of the regression run. The results from this enable us to hopefully allocate the test into the correct run (flaky or defect).
In the future we hope to automate the process of allocating the tests into the correct run.