Originally published at: LittleBITS: TidBITS Website and App Connectivity Issues Resolved - TidBITS
On 17 September 2024, we moved our Internet infrastructure to Cloudways, as I announced in “LittleBITS: Main TidBITS Server Moving This Week” (16 September 2024). Although we hoped things would go smoothly, there were inevitable hiccups. Most were behind the scenes, but readers encountered a few that impacted the TidBITS website and Matt Neuburg’s TidBITS News app. We’ve dealt with those issues now, so I wanted to explain for those who might have noticed problems.
Resolving the TidBITS Home Page Loading Error
On the morning of Saturday, 21 September 2024, a few days after we flipped the switch to direct traffic to the Cloudways-hosted version of our website, visitors started seeing a WordPress error when loading the home page. Article pages and the WordPress administrative interface were unaffected. Luckily, our developer was available, and within 20 minutes, he had identified and fixed the problem.
ImunifyAV, a malware scanner that’s part of the Imunify360 security package that Cloudways had recently installed, caused the problem. Even before we had taken the Cloudways version of the site live, ImunifyAV claimed to have identified malware on our site. It wouldn’t provide any details unless I subscribed, a sketchy marketing move for which Cloudways later apologized. It only costs $4 per month, so I reluctantly signed up, only to be told the reason was the cryptic error code “CMW-URL-23142.” The support rep said CMW translated to “client malware,” and that URL meant malicious code injected by URL, even though that doesn’t appear in the ImunifyAV documentation.
Regardless, the answer turned out to be that it was a false positive. I thought ImunifyAV was running only in scanning mode, but it decided to “clean” the “malware” it had detected. Removing the offending lines corrupted the surrounding PHP code. When our developer tried to use ImunifyAV’s option to revert, it gave him another “cleaned” version that was no better, so he reverted to a different backup. That fixed the issue and eliminated the error from the home page. I canceled my ImunifyAV subscription immediately.
Fixing the TidBITS News App Feed Download Failure
A few weeks later, on 11 October 2024, I started to get reports that Matt Neuburg’s TidBITS News app wasn’t staying up to date, reporting, “Error: Failed to download latest TidBITS feed.” The TidBITS News app relies on a custom RSS feed that had worked fine since the site move, and we initially couldn’t figure out what might have changed. Eventually, we contacted Matt to see if he could shed any light on the situation, and once he returned from a trip, we dove into the troubleshooting.
While working with Matt, he questioned what the point of Cloudflare was. I explained that Cloudflare is a content delivery network that improves website performance by serving cached copies of our content, handling about 93% of our traffic this way. That explanation caused me to wonder if caching was related to our problem, so I set up a Cloudflare rule to bypass the cache for the TidBITS News app’s feed URL. That resolved the problem, although we wouldn’t know why until the next issue cropped up.
Troubleshooting the “Please Wait” Error Page
On 26 October 2024, a few more days later, I started to receive sporadic reports of people not being able to load the TidBITS website, instead receiving a white screen with the text “Please wait while your request is being verified…”
Especially odd was the fact that the problem often afflicted only a particular browser or device. Nearly everyone who ran into it could load the site in another browser or device, but there was no consistency to the reports. Safari would work where Chrome didn’t for one person, whereas the next could load the site in Chrome but not Safari.
When I took the problem to Cloudways, the support rep said it was because our Cloudflare caching settings allowed users’ browsers to cache files, so I had to change the Browser Cache TTL setting to “Respect Existing Headers” so browsers would receive headers set by Cloudways or Imunify360. Unfortunately, the error persisted.
Eventually, I lucked out by receiving reports on the MacAdmins Slack from Alan West, Brad Chapman, and my friend Steve Sorbo, an Apple consultant in Seattle. With their tech-savvy testing help, I determined that the problem affected only www.tidbits.com URLs, not our currently canonical tidbits.com URLs. However, plenty of people use bookmarks created years ago, so the www.tidbits.com URL remains widespread and should always redirect seamlessly to tidbits.com.
After a few more hours and irritable messages to Cloudways support, I remembered that when we were troubleshooting the TidBITS News app problem, Matt had noted that the www.tidbits.com version of the feed URL worked when the tidbits.com version did not. Since that issue had been related to Cloudflare caching, I guessed wildly and cleared Cloudflare’s cache.
Amazingly, that enabled the “Respect Existing Headers” setting to take effect and eliminated the problem with www.tidbits.com URLs. When I reported my success, Cloudways support responded by telling me to do what I had just done and explaining that the previous settings had created a redirect loop—where Cloudflare and Imunify360 were repeatedly bouncing requests back and forth instead of serving the page.
As a welcome side effect, I was able to turn off the Cloudflare rule I had set up for the TidBITS News app feed. The problem revolved around the caching conflict between Cloudflare and Imunify360, not the difference between tidbits.com and www.tidbits.com.
Reserving Judgement on Cloudways
As you can probably tell, I’m not very happy with Cloudways right now. Although the company’s support is reasonably quick to respond and effective in the end, they often don’t carefully read what I’ve written and thus fail to answer questions completely or solve problems fully.
To be fair, I have to take some responsibility for these problems. Cloudways did alert all customers to the addition of the ImunifyAV malware scanning and Imunify360 firewall. Because we’re so new with Cloudways and because it’s common for service providers to send administrative messages about features that have no effect on existing setups, I failed to internalize that these changes might impact our site. Had I done that, I might at least have been less surprised when the problems cropped up.
Despite my frustration with these issues, I can’t fault Cloudways for wanting to improve security with Imunify360. Internet servers are under constant attack these days, and anyone running a public site must make usability compromises to prevent it from being overwhelmed.
For example, over the past year, we’ve struggled with spambots creating thousands of fake accounts on our site. I managed to slow them down for a while by installing a user verification plug-in and blocking hundreds of IP addresses in non-English-speaking countries. However, just before we moved to Cloudways, the spambots began using what I assume was a different botnet to bypass my blocks. Since then, I’ve deleted at least a thousand more unverified accounts, although the rate has slowed in recent weeks.