User:Daveh/Dev Guide

General Things

• Security

• Use the usual precautions regarding your login passwords (combination of letters, upper/lower case, numbers, symbols, +10 characters, don't reuse passwords from other sites, keep password secure, etc...).
• All servers will ban any IP that has 3 failed login attempts within a short period. If you get locked out or lose your password let the server admin know.

Servers

• Content1

• Content server that serves the Wiki pages, the forums, and a few other sites. Currently the primary (and only) content server for the Wiki.
• Runs: Apache
• Sites: Wiki (content1.uesp.net), forums.uesp.net, blog.uesp.net, ocp.uesp.net, eqwiki.net, dungeonhack.uesp.net (defunct)
• Request Rate: 5-20 requests/sec

• Content2

• Content server that serves the Wiki pages. Currently out of use due to intermittent connection issue.
• Runs: Apache
• Sites: Wiki (content2.uesp.net)
• Request Rate: 0 requests/sec

• Content3

• A extra server for a variety of things including minor site hosting, development, test, and backups.
• Runs: Apache, MySQL, Zabbix (monitoring)
• Sites: Wiki (content3.uesp.net), dave.uesp.net, monitor.uesp.net (Zabbix), mobile.uesp.net (temporarily), emergentgamedesign.com, viud.net (empty)
• Request Rate: <1 requests/sec (very low)

• Files1

• Holds the Wiki images which are shared to all content servers with NFS. Serves most static content for the Wiki and map images as well as the memcached server.
• Runs: Lighttpd, Memcached, Minecraft (sometimes)
• Sites: images.uesp.net, skins.uesp.net, maps.uesp.net
• Request Rate: 50-200 requests/sec

• Db1

• The primary (master) database which receives all writes for the Wiki and other sites. The database is replicated to all slaves from here (db2, content3, backup1).
• Runs: MySQL
• Sites: none
• Request Rate: 50-400 queries/sec

• Db2

• The main slave database which receives all reads for the Wiki (is not currently used by other sites or apps).
• Runs: MySQL
• Sites: none
• Request Rate: 100-600 queries/sec

• Squid1

• The main access point for the Wiki this serves the majority of web requests and passes all other requests onto the content servers.
• Runs: Squid
• Sites: www.uesp.net
• Request Rate: 20-100 requests/sec

• Squid2

• An extra server that sits outside of the private cluster and is current unused.
• Runs: Squid
• Sites: none
• Request Rate: 0 requests/sec

• Backup1

• An offsite server for backup purposes (not currently accessible from the Internet).
• Runs: MySQL
• Sites: none
• Request Rate: N/A

• Switch/Firewall

• There is a switch and firewall that sits between the Internet and the private cluster (content1/2/3, db1/2, squid1, files1).

Wiki

• The wiki is the primary source of traffic on the site with around 90% of all requests.
• There is enough traffic that the system can be relatively unstable. For example if something invalidates a large portion of any caches in the system it can noticeable impact the site for a while. Care should always be taken when editing the live site in any way. Testing all changes on the dev server (content3) is strongly recommended.
• Setup

• The wiki "lives" on the content servers in /home/uesp/www/w.
• The LocalSettings.php file is protected reading from anonymous or general users for security purposes.
• The wiki images are shared via NFS from files1.uesp.net via a path mounted under /mnt/. A local copy of images are copied on each content servers but there is no automatic fail over should the NFS share break.
• The wiki uses the memcached server hosted on files1. If this breaks the wiki will be severely impacted.
• The primary write database is db1 with db2 being the main read database. This can change if there are database server issues.
• The file cache is disabled as it causes issues when used in conjunction with the Squid cache.
• The LocalSettings.php files on each content server should be identical (barring a minor differences). Similarly the Wiki PHP files should be identical. Differences between content servers can cause strange issues.

Database

• The primary database (db1) is acting as the Master with all other database servers (db2, content3, backup1) acting as individual Slaves.
• The primary database can, and has, been switched between db1 and db2 when required. Typically this is due to issues on db1 which requires that it be taken down for maintenance.
• Content3 is generally only used for development and backup but but can be used for the live site if needed. Note that db2 is a significantly more powerful server. Db1 and content3 are the same but content3 is running other things than MySQL and is does not have an optimized configuration.
• All writes occur on db1 and most reads on db2. Reads can be split as needed in the Wiki configuration.
• db2 will not be used for reads if the database replication breaks or lags more than a few minutes.
• If replication breaks it can usually be manually repaired by either skipping a duplicated query or manually repairing the data. In the worst case a complete database backup from db1 will have to be reloaded on the slave to fix replication. This is why an occasional complete backup on db1 is important to have.
• Occasional checks on the slaves should be made to ensure the slave data is the same as the master data. This has not been performed in some time.
• All databases use a different user/password and no application uses the root user. Some applications further split between a read and write users.

Cache

There are a variety of caching layers used in the system in order to improve performance:

• Squid - The Squid cache is the first layer as it receives all web requests for the Wiki. It currently has a hit rate averaging 85%.
• Memcached - The Wiki content servers use the memcached server on files1 to hold various data. Note that this cache is RAM only so if you restart files1 or the memcached server it starts from nothing. Typical hit rate here is 95%.
• PHP Opcode Cache - The content servers all run eAccelerator which caches the compiled PHP byte code which increases the speed it can run at (typically around twice as fast). Hit rate here should be 100% as the cache is large enough to hold all PHP files.
• MySQL Query Cache - The MySQL instances on db1/db2 have the query cache enabled that lets the results from identical queries be saved for future use. Hit rate is around 95%.
• File Cache - The Wiki file cache is currently disabled due to issues using it with the Squid cache.
• Parser/Main/Message Cache - These Wiki caches are also currently disabled due to some issues. Further investigation is needed to enable these and find the cause of the issues.

Squid

• The Squid setup is a pretty standard setup with MediaWiki to cache incoming requests.
• It acts as a load balancer to all the content servers. Note that currently only content1 is in use. The Squid configuration can be edited and Squid restarted to take servers in and out of rotation with no service interruption (/etc/init.d/squid restart).
• The default Squid cache format does not work well for a high volume site like the UESP. After a while the file system daemon begins to lock up and cause issues. The fix for this is to use the __?___ cache format which has worked without any notable issues.
• Some modifications have been made to permit Squid to cache certain queries (like getmaplocs.php?x=...).
• Squid has been manually patched with the X-Vary-Options addition to permit it to work correctly with Google Analytics and other third party sites that use cookies.

Apache

• The PHP session path is pointed to the NFS share on /mnt/ so that sessions are shared between all content servers. This prevents issues from users being randomly logged out when Squid decides to send them to a different content server when load balancing.

Monitoring

• See Status for a variety of methods to view the status of the servers. The [1] Zabbix service has the most information and can be viewed by anyone (ignore the login and click the Monitoring tab in the upper left).

Developing

• The Eclipse IDE can be setup to use your ssh login to edit PHP files remotely (TODO: setup walkthrough).
• Content3 (will be) setup with a development environment using a local copy of the Wiki database that can be used for testing code changes without affecting the main site.
• Dev Server

• Content3 hosts the dev Wiki in the local path: /home/uespdev/www/w/
• Website URL is: dev.uesp.net/wiki/Main_Page
• Uses a copy of the Wiki database hosted on content3. Copy is currently controlled manually (no automatic updates).
• Images are a local copy in: /home/uespdev/www/w/images/.
• Various options are tweaked so the dev wiki is self-contained and isolated from the main wiki (Squid turned off, memcached turned off, etc...).

File Locations

• All Servers

• User Directory: /home/
• Logs: /var/log/
• Startup Scripts: /etc/init.d/
• Cron Scripts: /etc/cron.*/
• Local Backups: /home/backup/
• UESP Scripts: /uesp-setup/

• Content Servers

• Apache Config: /etc/httpd/
• PHP Config: /etc/php.ini
• Wiki Directory: /home/uesp/www/w/
• Shared NFS Mounts: /mnt/
• Local Copy of Shared NFS: /mntcopy/

• MySQL Servers

• Config: /etc/my.cnf
• Data Path: /var/lib/mysql/

• Squid Servers

• Config: /etc/squid/
• Cache Path: ?

• Files1

• Config: /etc/lighttpd/
• Shared Files: /shared/

Useful Commands

• General Stuff

• List File Details: ls -al
• View Process List: top
• View Memory Info: free -m
• VM Info: vmstat
• Free Disk Space: df -h
• Search for Process: pgrep <process>
• Network Info: netstat -an
• Online Shell User List: users
• Local Editor: vi
• Find App in Path: which <app>

• Yum Package Management

• Find App: yum find <package>
• Install App: yum install <package>

• Remote Copy

• Copy Remote Files: scp user@server1.uesp.net:/path/to/files/* /local/path/
• Sync Local/Remote Files: rsync -avz user@server1.uesp.net:/path/to/files/* /local/path/

• Security

• File Ownership: chown user:group filename
• File Permissions: chmod A+B filename

• A = a(all), u(user), g(group), o(others)
• B = r(read), w(write), x(execute)