Funny Javascript News Fader Experience

Over at the Early Retirement Forum, for which I assist in maintaining the forum software, I’ve been adding RSS icons and “Add to [My Yahoo!, Google, Bloglines, My MSN, etc.]” icons and working on getting a sensible RSS link and info system laid out. I have a news fader/ticker item that lists some of the icons and points to more RSS feed info. After collecting a number of icons in the images directory I decided to make and rss subfolder and move the images there so as not to clutter up the site owner’s images directory with RSS icons.

After moving the images I updated all my image links and made a mental note to check the log for 404 erros in the next day or two. The next day I checked, and one IP was creating three 404 errors every 30 seconds looking for the old image locations! Yikes! I tracked which user it was, and it’s a user that I know uses RSS but is not destructive. But there were no RSS feed requests associated with the image requests, so my first conclusion that he’s requesting RSS feeds to often was not correct. I made symlinks for the images in their original locations to stop the 404 errors and hopefully get the user agent, IE6 on WinXP, to cache the icons and quit requesting them.

I figured it out after checking the referer fields: The user must’ve left his browser open to the forum index before I made the image location changes. So his browser is staying open unattended all day while the news fader rotates through the messages and links to the images’ old addresses every 30 seconds or so. That’s why the images are being requested while he’s not hitting page views and not showing up in Who’s Online.

Normally this wouldn’t be a problem because the browser will cache the images and not reload them. But every now and then it will check to see if the image changed, but in this case it found the image missing, so after that it re-checked every time the image was requested, which was every time the image came up in the news fader rotation.

Now that I’ve symlinked the images back to their original location the user’s browser should cache the images and quit requesting them every time they come up on the fader. Next time he gets on the forum or closes the browser the issue will fix itself and I can take away the symlinks.

I love figuring stuff out. Note how some of the information seemed relevant at first but wasn’t: the icons were about RSS, and the IP belonged to a user known to use RSS, but it turns out that had nothing to do with the issue. On the other hand I guess we’re lucky it wasn’t an anonymous user who left his browser open or I might’ve assumed he was misbehaving somehow and banned his IP telling him he’s making too many requests when from his point of view he did nothing of the sort.


Shared and Concurrent Drupal Sites with Symlinking

I’m currently running Drupal 4.6.6, Drupal 4.7 RC3 and CivicSpace 0.8.3 . I preferred to have one set of source files, but I wanted each site (Apache vhost) to have its own root folder.

So, with some symlink magic, here’s what I do. Each version of Drupal has its own directory under my servers web root. I have the following directories and symlinks:

  • drupal-4.6.6
  • drupal-4.7.0-rc3
  • civicspace-0.8.3
  • drupal-4.6 -> drupal-4.6.6
  • drupal-4.7 -> drupal-4.7.0-rc3
  • civicspace -> civicspace-0.8.3

Each vhost has its own DocumentRoot with the following symlinks, substituting drupal-4.6 or civicspace for drupal-4.7 as appropriate:

  • cron.php -> ../drupal-4.7/cron.php
  • database -> ../drupal-4.7/database
  • includes -> ../drupal-4.7/includes
  • index.php -> ../drupal-4.7/index.php
  • misc -> ../drupal-4.7/misc
  • modules -> ../drupal-4.7/modules
  • scripts -> ../drupal-4.7/scripts
  • sites -> ../drupal-4.7/sites
  • themes -> ../drupal-4.7/themes
  • update.php -> ../drupal-4.7/update.php
  • xmlrpc.php -> ../drupal-4.7/xmlrpc.php

Each vhost gets its own files/ directory, robots.txt, .htaccess and favicon.ico files.

Now, when the next version (e.g. 4.7.1) is released I can unpack the files, copy the sites folder from the previous version, and then update the version symlink (drupal-4.7 -> drupal-4.7.1) and run the update.php script.

If I want to change a particular site from Drupal 4.6 or CivicSpace 0.8.3 to Drupal 4.7 I update the symlinks in the DocumentRoot to point to the 4.7 series, copy the appropriate site config file over and run the update.php script.

This is working well so far, but in my recent “Memory Hogging” blog I mention that having too many modules installed (even if not activated) make the admin/modules screen take up tons of RAM. My sites have different modules needs, so I think I’m going to give each vhost its own modules directory and then symlink the modules under that. This way I’ll have only one module version per drupal version in my filesystem and be able to remove modules I know don’t need from individual sites.


Memory Hogging

I’ve been hitting the limit of my VPS’s (virtual private server) 256mb RAM limit since installing CivicSpace, and I keep adjusting the php limit to try to avoid having forked processes fail, but then CivicSpace will fail on certain admin pages.

I now understand that when looking at the module activate/deactivate page it loads *every* module installed, even if it’s not activated. (For other pages it only loads activated modules.) Since CivicSpace includes so many modules, this almost guarantees I’m going to run out of php memory if I have the limit set at 20mb or 16mb, and that’s about the same range where I bump my head on the server privvmpages limit given my current configuration.

I don’t think I’ve had this problem with Drupal yet, but I probably will as I keep adding modules.

As a temporary fix I’m going to remove modules that aren’t used and aren’t likely to be used. For a more permanent fix I’m going to pay more to get a higher RAM configuration. Instead of just picking 384mb or 512mb I’d like to look into how I want Apache, MySQL and PHP tuned and figure out how much RAM should be dedicated to each. I might also setup my own test box and run stress tests on Drupal with various configurations. Then I’ll know my RAM target.


Drupal Module Quick Review

BBCode: (Input format / filter.) I love this module. Most of my intended audience is familiar with BBSes that use BBCode. With auto-url recognition and line breaks this is almost perfect for my needs. Perfect would include formatting buttons. Note that it also allows image links. (Images are so complicated in Drupal, but that’s another topic…)

BBCode_wysiwyg: (Formatting buttons.) Sounded like a great addition to the BBCode module, but it’s just 4 pushbuttons that don’t work as the buttons on the forums. I quickly turned it off. phpBB is GPL; I wonder how hard it would be to use their formatting toolbuttons for Drupal? UPDATE: I looked at phpBB’s form, and it’s also just html pushbuttons. I had remembered it being fancier.

Textile: (Input format / filter, included with CivicSpace.) Cute idea. *This text bolded* _this text underlined_ ; simple markup to produce valid xhtml. I especially like the links: “link text”:http://URL . Two problems: 1: it’s different than what my audience is used to. 2: you have to use white space to separate the markup from other text, so a sentence ending in bold, for example, would have a space before the period. Or comma, etc. That bugs me. I converted all content to BBCode and disabled Textile.

Short note on altering input formats: While playing with input formats I wound up with existing content using several different types of input formats. I wanted to simplify choices for users to BBCode or Plain Text (yeah, it’s a tad redundant), but in doing so it turns out the users can’t edit content created in that input format! A node administrator has to edit each bit of content and change the input format (and formatting, if necessary) to allow the users to edit their content again. (This is only if the users can’t currently use the input format the content was originally created in.) Moral of the story: Decide early in your site history what input options you want to use. Alternately offer tons of input types, but my users aren’t generally tech savvy and confuse with too many choices.

tagadelic: Very nifty. It lists your categories sorted as you want, but categories with more nodes in them have a bigger font indicating more content. I’ve only peeked at this module a little bit. At first it seemed like it shows all terms for all vocabularies, but then I noticed it created a nested menu item with links for each vocabulary. I wasn’t ready for that to be there but couldn’t disable the stupid thing! I got clever and moved it to my Secondary Links menu (Drupal 4.7) which I don’t display on my site.

votingapi: I just got this. It doesn’t do anything on its own, but it is the power behind simple voting and latest and greatest. After upgrading to Drupal 4.7 I played with free tagging taxonomies and was disappointed to learn you can’t have users (or guests) apply tags to existing content (a la rating/moderating/evaluating). I think I can work up something with votingapi to do this, though.

TinyMCE: (Formatting toolbar.) This thing and I got off to a bad start and just didn’t hit it off. I turned it off pretty quickly and haven’t tried it since. It reformatted my text box and removed all line/paragraph breaks in the box and in the output. I tried this as one of the first things I did with CivicSpace, and now that I know more about input formatting I may have to give this a try again as I’d really like some basic formatting buttons for my posters. UPDATE: I’m highly suspicious of an HTML editor, though…hackers could bypass the editor in a hand-crafted POST request. I’m much more comfortable using BBCode and escaping all HTML. Besides, I don’t really care for WYSIWYG; I just want an easy-to-use toolbar.


CivicSpace 0.8.3 Oddities

Although a relative newbie, I have some observations on CivicSpace 0.8.3 as compared to a base Drupal install:

  • The “plain text” input filter allows unfiltered html!
  • The bug list isn’t closely monitored. I submitted the unfiltered html issue to CivicSpace as a bug, and 4 1/2 days later it is still unassigned and has no replies.
  • The zadministration panel has a cute front page but mucks up the administration navigation menu in a way that’s confusing to me, different from Drupal (and therefore Drupal documentation and discussion) and when you turn off the zadministration module your administer settings are scattered amongst the top level of your navigation menu; in other words you don’t go back to Drupal normal admin when disabling zadministration. EDIT Dec 20, 2006: I just noticed how to avoid this problem when disabling zadministration: Before disabling the zadmin module, from the zadmin panel, click “Activate/Deactivate this page” and then push the “Deactivate” button. Then the admin menus return to their Drupal state and you can safely disable the zadmin module.
  • CivicSpace seems to be a memory hog compared to Drupal. CivicSpace recommends setting php.ini to use 24M of memory, but on my 256M virtual server I keep running out of VM pages when I do that. I’ve tried different settings and had problems at 8M and 12M; I’m at 16M now but I haven’t had a problem with a Drupal site at 8M with the same modules enabled yet. I think the enabled statistics module may be to blame, but I’m not sure yet. (I have CivicCRM turned off. I could also probably tune apache and MySql to not use as much RAM, but I haven’t tried that yet.)
  • After upgrading CivicSpace to Drupal 4.7 rc3 with a mix of Drupal 4.7 and cvs modules, the “my profile” page has some odd characters on it.

Upgrading CivicSpace 0.8.3 to Drupal 4.7 rc3

These past two or three weeks I’ve been working hard on figuring out CivicSpace / Drupal. I’m ditching CivicSpace 0.8.3–based on Drupal 4.6–and going with Drupal 4.7 rc3.

Drupal 4.7 is very new. In fact it hasn’t been officially released yet. But there are enough changes I think it’s important to go ahead and switch now while my websites are young and shy on content. The big feature for me is the free tagging additions to taxonomy. I don’t fully understand these yet but have some ideas on how I want to use them. As I understand it, the backported free tagging / folksonomy patches for 4.6 are sufficiently different from 4.7 as to cause upgraded headaches.

At first CivicSpace sounded like an easy-to-install Drupal distribution that included a robust default configuration with popular modules. I had used Drupal 4.6, but hadn’t quite figured it out, and CivicSpace seemed slicker. Perhaps it helped; I don’t know. But I’ve already outgrown the need for the preselected modules. As far as the easy install, it was easy, but I didn’t have a problem installing Drupal in the first place. As I learned to use CivicSpace I decided I didn’t like their “zadministration” modified administration menu, didn’t use the features (CivicCRM) that set CivicSpace apart from Drupal plus modules and started to suspect that CivicSpace is enough different from Drupal that “upgrading” to Drupal could be a problem, especially with established content.

So, on the the upgrade. I installed the drupal 4.7 files in a new directory, copied my files/ directory and configuration directory over and checked the .htaccess and robots.txt files. I downloaded and installed the modules that were enabled in CivicSpace but not included with Drupal. Some modules have a 4.7 version, some I had to get the cvs build. Actually I left out the zadministration panel module because I couldn’t find it on Drupal’s site, and I hate it, anyway.

EDIT Dec 20, 2006: I just noticed how to avoid the administration menu problem when disabling zadministration: Before disabling the zadmin module, from the zadmin panel, click “Activate/Deactivate this page” and then push the “Deactivate” button. Then the admin menus return to their Drupal state and you can safely disable the zadmin module. (/EDIT)

Drupal 4.6/CivicSpace don’t have a “site maintenance” mode; I wanted to prevent possible changes to the database during the upgrade. I found suggestions on using conditional apache redirects, but it occurred to me I could just remove authenticated users’ permission to access content. To make a friendly message, I created a new block for the top of the left column explaining the site was briefly down for maintenance. After enabling that and disabling “access content” under the node section of the access control page everybody except user #1 gets “Access Denied”, the login box and my site mx message block. I patted myself on the back for being so clever and knowing how to do this.

I backed up the databse. The files hadn’t changed since the daily backup, and I didn’t intend to delete or overwrite anything, so I didn’t do any specific file backup. I logged in as user #1 as per the upgrade instructions. I disabled the zadministration module because I wasn’t going to use it in 4.7. I used the unix “mv” command to move the existing install to an old/ directory and move the new directory in its place. Already logged in as user #1, I browsed to /update.php . Drupal 4.7 is pretty slick in that it has upgrade hooks for its modules, meaning you don’t need to update modules individually. This screen handles upgrading the datbase for the system and all hooked modules.

That was easy, and then the site worked fine. I poked around a bit and then reenabled user access to content and disabled the site mx message block. There were a couple of oddities at first, but they cleared and were apparently caching issues. Only two issues remained: the admin menu was messed up due to disabling zadministration; the primary links were missing (I don’t use secondary).

EDIT Dec 20, 2006: The zadministration menu issues in the previous and following paragraphs are avoidable by “deactivating” zadmin before disabling the module. See my earlier edit above for details. I haven’t upgraded a deactivated zadmin site yet, so I’m not sure yet if the Primary Links problem will still happen.(/EDIT)

These issues are actually the same. Drupal 4.7’s main menu is called “Primary Links” and is propogated to the Primary Links area on the theme as well. But due to the zadministration influence your “upgraded” menu doesn’t fit the scheme. I had done a Drupal 4.6->4.7 upgrade before the CivicSpace upgrade, so I knew what the menu should look like. Rather than rebuild it by hand I used mysqldump and mysql to dump the menu table from the “good” site’s menu and import it into my upgraded CS site, overwriting the existing structure. (Lots of backups and careful typing here, of course.) It worked great! I then had to delete an item I had added to the other site’s menu and manually add links to my aggregator sources. Interestingly it automatically deleted the invalid aggregator source links that were imported from the other board. After this my navigation menu, administration menu and Primary Links along the top of the page were working spiffy and the menu adapted to new modules I enabled. UPDATE: Drupal 4.7 doesn’t add menu items under aggregator/sources anymore; that’s probably part of why I had to manually re-add the menu items after doing my menu trick. Come to think of it, I think I’ll take the aggregator/sources subitems back off again as I have too many feeds to make sense of; I’ll set up the categories instead.

The only two remaining glitches I know of are that the profile page is slightly garbled and the “1,2,3 next page” links are squished together for some reason. The squished links are on my 4.6->4.7 upgrade site, too, so I don’t know if that’s an rc3 issue yet or an upgrade artifact. UPDATED: The “squished links” were apparently also a caching issue (cached CSS file?). It looks fine now for both the 4.6->4.7 site and the CivicSpace->Drupal 4.7 site, and I haven’t changed anything.


Cute “Add to …” Icons for Web RSS/News Readers

I’ve been geeking out on RSS stuff lately. I was aware of its existence before but didn’t have a lot of interest in it. Due to some web projects I’m doing I have a new interest in syndicating via RSS, and that’s what started this RSS geekfest of mine.

This entry is mainly to list the URLs for site instructions on those cute one-click “Add to…” buttons to add RSS feeds to their web-based news reader.

Heh, this one does many of the others for you: