LittleBITS: TidBITS Website and App Connectivity Issues Resolved

Originally published at: LittleBITS: TidBITS Website and App Connectivity Issues Resolved - TidBITS

On 17 September 2024, we moved our Internet infrastructure to Cloudways, as I announced in “LittleBITS: Main TidBITS Server Moving This Week” (16 September 2024). Although we hoped things would go smoothly, there were inevitable hiccups. Most were behind the scenes, but readers encountered a few that impacted the TidBITS website and Matt Neuburg’s TidBITS News app. We’ve dealt with those issues now, so I wanted to explain for those who might have noticed problems.

Resolving the TidBITS Home Page Loading Error

On the morning of Saturday, 21 September 2024, a few days after we flipped the switch to direct traffic to the Cloudways-hosted version of our website, visitors started seeing a WordPress error when loading the home page. Article pages and the WordPress administrative interface were unaffected. Luckily, our developer was available, and within 20 minutes, he had identified and fixed the problem.

WordPress critical error

ImunifyAV, a malware scanner that’s part of the Imunify360 security package that Cloudways had recently installed, caused the problem. Even before we had taken the Cloudways version of the site live, ImunifyAV claimed to have identified malware on our site. It wouldn’t provide any details unless I subscribed, a sketchy marketing move for which Cloudways later apologized. It only costs $4 per month, so I reluctantly signed up, only to be told the reason was the cryptic error code “CMW-URL-23142.” The support rep said CMW translated to “client malware,” and that URL meant malicious code injected by URL, even though that doesn’t appear in the ImunifyAV documentation.

Regardless, the answer turned out to be that it was a false positive. I thought ImunifyAV was running only in scanning mode, but it decided to “clean” the “malware” it had detected. Removing the offending lines corrupted the surrounding PHP code. When our developer tried to use ImunifyAV’s option to revert, it gave him another “cleaned” version that was no better, so he reverted to a different backup. That fixed the issue and eliminated the error from the home page. I canceled my ImunifyAV subscription immediately.

Fixing the TidBITS News App Feed Download Failure

A few weeks later, on 11 October 2024, I started to get reports that Matt Neuburg’s TidBITS News app wasn’t staying up to date, reporting, “Error: Failed to download latest TidBITS feed.” The TidBITS News app relies on a custom RSS feed that had worked fine since the site move, and we initially couldn’t figure out what might have changed. Eventually, we contacted Matt to see if he could shed any light on the situation, and once he returned from a trip, we dove into the troubleshooting.TidBITS News failure to download feed error

While working with Matt, he questioned what the point of Cloudflare was. I explained that Cloudflare is a content delivery network that improves website performance by serving cached copies of our content, handling about 93% of our traffic this way. That explanation caused me to wonder if caching was related to our problem, so I set up a Cloudflare rule to bypass the cache for the TidBITS News app’s feed URL. That resolved the problem, although we wouldn’t know why until the next issue cropped up.

Troubleshooting the “Please Wait” Error Page

On 26 October 2024, a few more days later, I started to receive sporadic reports of people not being able to load the TidBITS website, instead receiving a white screen with the text “Please wait while your request is being verified…”

Especially odd was the fact that the problem often afflicted only a particular browser or device. Nearly everyone who ran into it could load the site in another browser or device, but there was no consistency to the reports. Safari would work where Chrome didn’t for one person, whereas the next could load the site in Chrome but not Safari.

Please wait error

When I took the problem to Cloudways, the support rep said it was because our Cloudflare caching settings allowed users’ browsers to cache files, so I had to change the Browser Cache TTL setting to “Respect Existing Headers” so browsers would receive headers set by Cloudways or Imunify360. Unfortunately, the error persisted.

Eventually, I lucked out by receiving reports on the MacAdmins Slack from Alan West, Brad Chapman, and my friend Steve Sorbo, an Apple consultant in Seattle. With their tech-savvy testing help, I determined that the problem affected only www.tidbits.com URLs, not our currently canonical tidbits.com URLs. However, plenty of people use bookmarks created years ago, so the www.tidbits.com URL remains widespread and should always redirect seamlessly to tidbits.com.

After a few more hours and irritable messages to Cloudways support, I remembered that when we were troubleshooting the TidBITS News app problem, Matt had noted that the www.tidbits.com version of the feed URL worked when the tidbits.com version did not. Since that issue had been related to Cloudflare caching, I guessed wildly and cleared Cloudflare’s cache.

Amazingly, that enabled the “Respect Existing Headers” setting to take effect and eliminated the problem with www.tidbits.com URLs. When I reported my success, Cloudways support responded by telling me to do what I had just done and explaining that the previous settings had created a redirect loop—where Cloudflare and Imunify360 were repeatedly bouncing requests back and forth instead of serving the page.

As a welcome side effect, I was able to turn off the Cloudflare rule I had set up for the TidBITS News app feed. The problem revolved around the caching conflict between Cloudflare and Imunify360, not the difference between tidbits.com and www.tidbits.com.

Reserving Judgement on Cloudways

As you can probably tell, I’m not very happy with Cloudways right now. Although the company’s support is reasonably quick to respond and effective in the end, they often don’t carefully read what I’ve written and thus fail to answer questions completely or solve problems fully.

To be fair, I have to take some responsibility for these problems. Cloudways did alert all customers to the addition of the ImunifyAV malware scanning and Imunify360 firewall. Because we’re so new with Cloudways and because it’s common for service providers to send administrative messages about features that have no effect on existing setups, I failed to internalize that these changes might impact our site. Had I done that, I might at least have been less surprised when the problems cropped up.

Despite my frustration with these issues, I can’t fault Cloudways for wanting to improve security with Imunify360. Internet servers are under constant attack these days, and anyone running a public site must make usability compromises to prevent it from being overwhelmed.

For example, over the past year, we’ve struggled with spambots creating thousands of fake accounts on our site. I managed to slow them down for a while by installing a user verification plug-in and blocking hundreds of IP addresses in non-English-speaking countries. However, just before we moved to Cloudways, the spambots began using what I assume was a different botnet to bypass my blocks. Since then, I’ve deleted at least a thousand more unverified accounts, although the rate has slowed in recent weeks.

4 Likes

While I appreciate the efforts described herein, which restored my TidBITS iOS application feed, as of this writing that feed has seized up again in the same fashion. It does not appear to be a problem with my connection, as I can access the website via a browser (the DuckDuckGo application in this instance) on the same device as I use the application on, a Fifth-Gen iPad Air running iPadOS 18.1. Neither quitting and restarting the application, nor shutting down and restarting the iPad, appears to have any effect. Please advise. Thank you.

1 Like

Sorry about that! In the “Once You Say Something in Public” category, as I was finalizing the TidBITS issue last night at 6 PM, we got hammered with an incredible traffic spike that took down the server.

Although the Cloudways chat support hasn’t worked too well for me so far, when I realized that the CPU usage for the server was pegged at 100%, I got in touch with them via chat. The support rep correctly diagnosed the problem as an insane bot attack, isolated a bunch of user agents that were hitting the server, and blocked them all. That resolved the problem, but at the cost of blocking legitimate traffic too. She gave me a list of what she blocked, and while some were obviously legitimate (like Discourse), I didn’t immediately recognize some others. She removed the legitimate ones I mentioned to her and let the rest stay blocked. Before I had to sign off to go to a meeting, she gave me the information I needed to edit .htaccess on our server so I could clean things up this morning.

Among those blocked user agents was CFNetwork, which looked familiar. When I heard from you and another person that the TidBITS News app was failing again, I double-checked, and indeed, CFNetwork is the user agent for apps using Apple APIs. Since using curl to load the TidBITS News feed worked (where the user-agent wouldn’t be from the app, I was confident that unblocking CFNetwork in .htaccess would resolve the problem, and it did.

So, as much as I’d like to say all’s well that ends well, I’m sure we haven’t seen the last of these sorts of problems. But the more I learn about the new system, the more I’ll be able to solve them quickly on my own.

1 Like