UESPWiki:Archive/CP UESP server problems

This is an archive of past UESPWiki:Community Portal discussions. Do not edit the contents of this page, except for maintenance such as updating links.

UESP server problems

As you all know, the site has been effectively dead for most of the last few days. If you are able to see this, it means that you've accessed the site during one of its brief periods of availability. At the moment, all that is happening is that the server is being restarted occasionally; this clears things up temporarily but does not address the underlying problem. After it is restarted there is a period of one to two hours during which it is possible to use the site. I don't have any control over when those restarts occur, nor do I have the ability to implement the more longterm solution that is needed to really fix things.

If you're interested in knowing how long you have during your current access window, you can check the server status and see how recently the server was restarted ("Server Uptime"). When that time reaches an hour, the site becomes barely functional; after two hours, it is nearly impossible to get a response from the site.

In the meantime, feel free to use the site as normal. Normal user edits are NOT responsible for what is happening; you viewing or editing a page will not make the site's availability deteriorate any faster than normal. Based on my assessment, the key problem is that a web crawler (used to collect information for a web search engine) is flooding the site with page requests, because our configuration file for web crawlers is not providing proper instructions to the web crawlers (trying to avoid too much techno-speak). --Nephele 14:30, 12 December 2006 (EST)

As announced on the news on the main page, a real fix to the problem has now been implemented (thanks Daveh!) Preliminary status reports show that the server seems to be doing much better now, so I think the server will stay available for more than two hours this time around. I'll definitely be continuing to keep an eye on things during the day, but I think everyone should be able to start using the Wiki as usual again! --Nephele 14:41, 13 December 2006 (EST)

Yes, the server has been having some problems again. Unfortunately, the server needs to be restarted every couple days in order to keep functioning smoothly; this has been the case for nearly a month now. Daveh is the only one who can do such restarts and he is currently on travel, so it took five days for the server to be restarted. I would desperately like to find a better long term solution to this problem, but at the moment there is none.

One temporary measure that would help is if those users who use the RSS feed could avoid using it for a couple weeks. Broken RSS connections are the main reason that the server becomes sluggish and eventually completely inaccessible. Although the major shutdown last weekend was caused by robots inundating the site, the more recent slowdowns are just due to occasional broken connections from regular users; the most common broken connections are when accessing the Recentchanges page using the RSS feed. Although the RSS feed is not fundamentally the problem, it is aggravating the underlying problems with the server. And with Daveh only intermittently available from now until January, any stop-gap measure that will keep the server semi-functional for longer needs to be considered.

I realize that those of you accessing the RSS feed probably rely upon it for UESP updates, and not using it will be a significant inconvenience. But I think it's possible that if everyone stops using RSS for the immediate future, the server will be able to stay up for much longer at a time, maybe even until January without any more restarts. So I sincerely hope that those of you who have been using the RSS feed will consider making this small sacrifice for the good of the site. (And for those of you wondering, no, I don't have the power to block the RSS feed and force this change, it has to be done voluntarily by the users at the moment). Thank you! --Nephele 11:13, 18 December 2006 (EST)

Is there any way to disable the RSS feeds? I checked quickly but didn't find any obvious setting for it. I won't have any internet access for 2 weeks or so during Christmas so need a way to ensure the RSS bug doesn't bring down the server during that time. -- DaveH 13:46, 18 December 2006 (EST)

Good question :) At the moment, all I see is a couple of settings that may make the RSS feeds less likely to misbehave. My gut feeling right now is that when it takes the server too long to respond with a RSS feed, the client gives up and that leads to the dropped connection. In which case, decreasing the amount of info that gets included in the RSS feed may help. So some settings that could help are:

• $wgFeedLimit, number of results to return, default value 50, decreasing (20? 10?) should make the data stream smaller
• $wgFeedDiffCutoff, maximum size of the diff results to return, default value 32768, again decreasing (5000?) should allow the server to respond more quickly
• $wgFeedCacheTimeout, default value 60. This is the one I understand the least, so it might be best to leave it as is. It sounds like increasing it would reduce the server workload.

I'll keep rooting around in the code, and post anything else that I see. --Nephele 15:22, 18 December 2006 (EST)

OK, one more possibility: set $wgFeedClasses = NULL;. This variable is described as "Available feeds objects"; hopefully setting it to NULL in LocalSettings.php will override the default setting in Defines.php. If this array is empty, any attempt to access the RSS or Atom feeds should result in the code immediately returning a 500 error message ("Internal Server Error, Unsupported feed type"). --Nephele 15:34, 18 December 2006 (EST)

When it rains, it pours.... There now seems to be a new UESP problem, which is completely unrelated to the previous problems (for those who are curious, the server itself seems to be doing quite well right now). It now seems to be very difficult to post any edits to the site. Pages can be accessed and read without problems, and some actions (such as previewing edits and deleting pages) seem to work without problems. I'm unable to do any real diagnosis of the problem, but I'm guessing there may be a problem with available hard disk space. The only evidence for this is that after deleting a handful of images that had been proposed for deletion, I was able to post an edit. Based on that assumption, I'm going to try to clean up unused images and purge thumbnails of rarely-used images. --Nephele 02:10, 24 December 2006 (EST)

It seems increasingly likely (to me, at least) that this is a problem caused by a shortage of hard disk space. Therefore, I'm requesting that nobody post any new images to the site until this can be fixed (which as with all other issues, is unlikely until January when Daveh returns from travel). I'll continue to try to make a bit of space by getting rid of redundant and unused images (and apologies for bypassing the proposed deletion process while doing this, but in my opinion the situation warrants it). --Nephele 02:36, 24 December 2006 (EST)

Yes, unfortunately the server filled up its hard drive which was causing the problem. It seems it has been keeping some 60GB of web server logs (ouch!) which I wasn't aware of until now. Things should be fine now as I've deleted a few GB of old backups and am in the process of cleaning up the old logs. Figures this happens a few days after I leave for the holidays. -- DaveH 15:24, 4 January 2007 (EST)

I am experiencing a very slow connection to the site right now. Is there a problem with the server again? --DrPhoton 03:59, 25 January 2007 (EST)

There is a new issue that has been cropping up lately: computers inundating UESP with requests in order to download the entire site. Last night a single IP address (corresponding to a location somewhere in China) was hitting the site. Every time I checked the server status, almost every available connection was being used by that one IP address, and every time I refreshed there was a new list of files that the IP address was requesting. The night before, the same thing happened, but it was a different IP address that night. At various points over the last two weeks I've noticed at least five different IP addresses doing this; each time, the server becomes pretty slow in responding. I've told Daveh about the problem, and it seems that he's tinkering with things right now. Fixing it may require adding some new capabilities to the server, so it can "throttle" individual IP addresses that try to monopolize excessive bandwidth. But the good news is that the problems that were happening over christmas seem to have been resolved! --Nephele 10:43, 25 January 2007 (EST)

Yes, as Nephele described the recent slowdowns are due to entirely different reasons than the previous site issues. Currently I'm just manually banning IPs that clog the site (currently only 2) but am looking into installing a mod_throttle or similar apache plugin that will automatically handle such bandwidth abuses.

If you happen to notice a particular site monopolizing connections just record its IP and let me know (e-mail or a talk page) since I cannot continually monitor the site and may miss them (technically I should be able to see them from the logs but I'm not sure exactly how atm). -- Daveh 12:44, 25 January 2007 (EST)