Alright!  As of tonight we should be completed with all the planned *.cerb4.com On-Demand migrations to the new network.

If you host with us and have a firewall in front of your POP3/IMAP account, ping your *.cerb4.com site and update your firewall with the new IP.

If you have a DNS alias (support.yoursite.com -> yoursite.cerb4.com), you should switch to using a “CNAME” record to your *.cerb4.com domain rather than an “A” record to a specific IP.  In the future you won’t have to update your IP if it changes.

Enjoy!

-Jeff@WGM

The RAID controller/storage on one of the Cerb4 On-Demand servers started throwing constant kernel panics this morning.  We decided to just quickly migrate the affected sites (about a dozen) to new hardware before we started our troubleshooting and hardware replacements.

As we decouple our On-Demand installations from specific hardware it will be increasingly common for IPs to change.  If you’re using a DNS “A” record to customize your domain, you should switch to a “CNAME” record that maps to your *.cerb4.com domain opposed to a specific IP.  That will transition cleanly as your site moves around on our network.  Feel free to drop us a note if you need help with that.

The downtime this morning for the affected sites was minimal but sporadic (due to reboots).  Everything should be moved over now (with the exception of a few historical attachments if your helpdesk is huge — those are still copying as I type this).

Hardware has no common courtesy ;) — but at least the Cerb4 design makes things easy to move around in minutes if the need arises.

-Jeff@WGM

So far we’re keeping a really good pace going in development through June 2008.  This is our 3rd release to the stable branch this month.  I’m enjoying getting these updates out to you guys much quicker, rather than working toward arbitrary “4.0 Release” or “4.1″ versions with months of hidden work behind them.

This latest stable release includes the functionality I blogged about going through QA testing on Monday.  There are several performance tweaks.  Support was also added for per-worker timezones and advanced fulltext searching on message content (booleans, boosting, wildcards, phrases).

You can view the full changelog in the forums.

Enjoy!

-Jeff@WGM

Things are moving pretty fast over here.  Busy day!

Quality Assurance (QA)

Mike & Joe spent the day doing a full QA on Build 639 so we can push the latest changes into the stable branch.  They found a couple minor things just by being incredibly comprehensive, but the fixes shouldn’t cause a significant delay.

Project Portal (JIRA)

Dan and I have spent part of the past couple work days cleaning up the project portal.  Today we vastly simplified the process of adding bugs and feature requests.  Most of the default fields that JIRA was prompting for weren’t actually used by our workflow.

We also removed the ‘priority’ field from public editing because it’s so arbitrary.  We replaced it with the 3-point agile priority system we use internally:

1. Must Have: Cerb4 is considered incomplete without this fix/request.  Most of the time we don’t use priority 1 for feature requests.  This helps ensure we’re tackling bugs before more glamorous and fun tasks writing new features.
2. Significant Value: These requests will bring more people to the project and will make the people who are already spending money with us even happier about their investment in our development.  We base a lot of ’significant value’ on what we’re hearing in feedback (and how often we’re hearing it).  We also factor in the votes.
3. Nice to Have: This is pretty much everything else.  Features in this category are done last — they aren’t holding anybody up and they won’t bring in significant new business.

Time Tracking

We’ve pulled together our dozens of notes on the “time tracking” feature requests and we’ve started development.

We have a new plugin for time tracking.  It currently adds a new tab to Display Ticket for keeping track of time slips related to specific tickets.  We have a global timer that floats in the top right of the application and counts up like a stopwatch.  You can play/pause/stop that timer.

It’s currently set up so you can start the timer as you view a new ticket.  Your time is then being logged as you read and reply to the message.  After you submit your reply, as usual, you’ll be able to stop the timer by clicking the “stop” button in the top right.  This will pop up a panel (like “peek” does everywhere) to allow you to enter comments about the time slip before it saves back to the ticket automatically.

We’ll allow time slips to be added to several objects, just like tasks work.  You’ll be able to do a quick report of all time slips over a date range.  You’ll be able to quickly mark whether an entry is billable or not.

There’s no guarantee that this won’t change significantly as development continues, but here’s a sneak peek:

Development since last week continues to focus on performance, stability, and usability before returning to new functionality; however, we have managed to sneak in one highly requested feature and several minor tweaks.

Performance

Performance is already great for most single helpdesk installations, but every day we’re monitoring hundreds of helpdesks from our Cerb4 On-Demand service.  We find several bottlenecks from deploying Cerb4 on that scale that will speed up all installations.

The big performance improvements pending QA testing for the next stable release are:

• SMTP connections are now persistent for functionality that sends a lot of mail; specifically e-mail notifications and auto-replies.  This is a lot more efficient.
• Message content can now be searched using MySQL’s FULLTEXT indexes, including phrases, booleans, boosting and wildcards.  Before we went MySQL4+ specific, our previous approach used the MUCH slower and less flexible “LIKE” syntax.
• Added support for persistent MySQL connections.
• Added more objects to the cache manager: worker preferences, ticket fields, group memberships.  This saves a lot of needless queries to the database from most pages.
• Fixed the cache manager when using disk-based caching (the default).  It was invalidating the caches too often, causing unnecessary I/O work and database queries.
• Improved how Devblocks (the platform) renders plugin-provided pages (which is basically all of them).  Less work required to render pages means faster load times.
• Drastically improved the performance of the nightly maintenance task.  We added a new “cron.maint” event that triggers after deleted tickets are purged.  It gives each plugin a chance to perform their own cleanup — which, for the main app, means cleaning up any orphaned data in the database (e.g. messages with deleted parent tickets, etc).  Previously the maintenance task was running an event for every deleted ticket.

New Functionality

• Timezone support has been added on a per-worker basis.  Workers can click “My Account” and select a timezone which will control date input and output.  This was badly needed to get the most out of our flexible date entry (”5pm” or “+2 hrs”, etc).  If I enter a date of “5pm” in Los Angeles time, another worker would see “8pm” in New York time.

This weekend we’ll be performing the second half of our server network overhaul for our Cerb4 On-Demand service.  This will affect all *.cerb4.com websites for a short time.

Here’s the plan:

• We’ll be updating all helpdesks to the latest stable build (that’s currently 638 if all goes well in QA for Weds).  There are a lot of optimizations in this build that came from monitoring our hosted helpdesk network under higher loads.  While already fast, the improvements should be noticeable all around.
• We’ll be migrating most helpdesks to our newer hardware: Dual Quad Core (8 cores) Harpertown servers, 6GB RAM, 500GB RAID-1.
• The new machines have off-RAID storage for local backups, which should fix the issues with brief slowdowns around nightly backups and maintenance (~10PM Pacific).  A current limitation of most of the existing machines is they’re in 1U rackmounts with limited drive slots.  We keep nightly backups on the local disks and we off-site 2x a week (Weds and Sat).  Local backups are kept uncompressed to not bog down the live machines.
• We’ll be balancing server populations according to resource usage and our monitoring.  This ensures everyone has plenty of performance and room to grow.
• With the increased memory and lower server populations we’ll be enabling APC (PHP compile cache) and memcached (shared memory cache) for most helpdesks.  We’ve seen average page generation times of 5-10ms on most of the highest traffic helpdesks we’ve been testing these options with.

We’ll try to get everyone upgraded and on the new hardware during the window this weekend.  It’s a lot of data to move around.  We’ll post another followup if we need to break the upgrade up over a second weekend.

Send us an e-mail and we’ll schedule you sooner if you’d like to have your hosted helpdesk moved earlier to take advantage of the ever-increasing performance.

Thanks!

-Jeff@WGM

Our return to flat-rate pricing has definitely been better received than our foray into per-worker licensing.  We’re glad to be back.

When we were having those pricing discussions, a lot of the same interesting ideas kept coming up.  So here’s what we’ve done:

On-Demand Trials

We have been hosting our “instant evaluations” on separate machines from our standard hosting service, but this has led to overloading and abuse.  Beyond that, people were just being dumped into a new installation without any introduction, and we didn’t have a good way to know who, of the thousands of people, would actually like to hear from us (and who was just taking a curious peek).  It ate a lot of time we could have been spending better.

So we’ve done two things to improve this:

• We improved our shared online demo so it has a constant influx of simulated tickets and spam.  We’ve created a much more believable (and instructive) sample environment of Groups, Buckets and Filters.
• We’re now offering a free 14 day trial period on our Cerb4 On-Demand (software as a service) offering.  This doesn’t just help people who were considering hosting with us; it also allows evaluators to test an instantly working installation on our servers so they can decide if they’d like to move forward with Cerb4 before investing in hardware and headaches (e.g. mailparse).  While the previous instant evaluations served this purpose, we’re now setting up all evaluations on our premium hardware and network instead of our spare evaluation and Q.A. machines.

Lease-to-Own Licenses (Standalone)

Many people, especially smaller web hosts, have been asking us for a way to break up the Cerb4 purchase price over a few payments.

We’ve added the ability to break a purchase up over 3, 6, and 12 months.  To curtail a needlessly delayed accounts receivable — and the risks and hassle associated with having to charge a credit card up to 12 times to get paid — we’ve added a small markup to the total cost of a license purchased on installments.  The fewer payments, the less of a difference over paying up front.  Still, it’ll be much more efficient to allow people to choose these options from the website opposed to having people contact us for a custom invoice every time.

On-Demand Prepayments

Our 20% discount on annual payments for the Cerb4 On-Demand service has been pretty popular.  We’ve also had people expressing interest in smaller discounts for shorter prepayment cycles.

We’ve added a 5% discount for 3 month prepayment, and a 10% discount for 6 month prepayment.

New Free Licenses

By far, the best way to spread the word about Cerberus Helpdesk over the years has been by offering a non-crippled free version — something that really allows people to use it with no strings attached.

We’ve had some minor criticism that our 4.0 free version, with its e-mail tagline on outgoing mail and cap of 3 groups, was betraying the history of a usable, entry-level free version.  On reflection, we agree.

So here’s the tradeoff.  At the end of a new installation, we now show a short survey that allows a new organization to quickly introduce themselves (and tell us what they’re looking for, and what they’re measuring as a successful helpdesk evaluation).  Anybody who fills out this survey will receive a registered license for 3 workers by e-mail for free.  This removes the group cap, and the Cerb4 advertising tagline at the bottom of outgoing e-mail.

This keeps our stream of constant feedback alive.  We’ll only contact people about Cerb4 and their experience with it.  Free users are still welcome to skip the survey and stay in stealth mode, but they’ll keep the other limitations.

Conclusion

We’re happy with these latest tweaks.  They’re a lot more utilitarian than changing the entire approach toward pricing every few months.

Keep the feedback coming!

We’re finally delivering on our goal of making these stable releases a lot more frequent.  This is the 7th update from user feedback in the stable branch for 2008, but it’s coming only a week after the last update (which was 20 days on the heels of the release before that).

I talked about Build 603’s major performance boost here on the blog; but we haven’t talked about the most recent improvements since then.

The primary focus lately has been on performance and usability.  We’re definitely still making progress on the major feature requests, like localization, time tracking, and reports.  It’s just more important to make sure we’re stopping to tidy things up every couple releases so we can keep up the same development agility.

We’ve still been getting the most insight on performance tuning and optimization from our network of on-demand helpdesks.  For our environment, the biggest speed improvements have come from moving the caches off the local disk and by enabling APC (a PHP compile cache).  Cerb4 and Devblocks have had memcached support for shared memory caching since early on, but until these recent builds we hadn’t been supporting truly distributed caches.  The recent builds also add prefix support to the cache manager so multiple installations can share a caching back-end (which is especially helpful in a hosting environment).

On the usability end, we had a couple nice improvements:

• We took all the feedback (mainly from the forums) about ticket comments and added the ability to display comments threaded into the ticket conversation (chronologically).
• We added a subtle visual indicator to elements of a ticket conversation: inbound, outbound, comments and notes.
• We improved the interface by softening up some stark lines between major elements (especially on Mail Overview or Ticket Display).  We added a consistent style to all the buttons in the application so they should look similar across browsers (and regardless of your OS theme).
• We streamlined the guided installer so it’s no longer asking new users to create Workers or Groups before they have an understanding of what those things mean.  The installer now creates some sample Groups and Buckets.  It also pre-configures group spam rules.  The first ticket for the helpdesk is now procedurally generated as the last step of the installation.  It shows up as a message from our development team welcoming people to the project and telling them what to do to get started.  Replying to that message from their Cerb4 will actually send a message right to our support team.  It’s an easy way to get help from inside the app, while also exploring how things work.
• We’ve added more keyboard shortcuts to Mail Overview and Display Ticket.  The possible shortcuts are now more context-sensitive when displaying the keys (it won’t show you shortcuts that won’t actually do anything for the current items).
• A “Read All” mode has been added to the Display Ticket screen, which will automatically expand every message and show the conversation chronologically.
• When assigning a Next Worker you can now also choose when to expire the assignment (such as at the end of your shift).  After the expiration, any worker will be able to handle the follow-up responses.  This still allows better efficiency during a shift where workers can handle their own short-term followup replies.

There are also a two dozen or so smaller feature requests and bug fixes.

You can read the full changelogs here:

http://www.cerb4.com/forums/forumdisplay.php?f=10

Enjoy!

Hey guys,

We’ve been meaning for a long time to upgrade the machine that our project Subversion repository is hosted on. The older machine started giving us some trouble this afternoon, so we went ahead and migrated our Subversion repositories to newer hardware.

Here are the instructions for updating your working copy to the new location:
http://wiki.cerb4.com/wiki/Changing_SVN_Repository_Locations

We also went ahead and set up support for tags and branches since we’re at the point with 4.0 we can start actually using them. In the instructions above you’ll be able to lock your installation to only get ’stable’ (post-QA tested) updates when you run ’svn update’. If you want to always be using the latest development versions there’s a note at the bottom of the docs.

Update: I should also note that ’stable’ is considered Build 622 as of the move. We’ve done our due dillgence testing the build, but you’ll probably get a conflict on the framework.config.php file after updating. If you aren’t familiar with fixing conflicts, the easiest way to clean this up is to copy your framework.config.php file to framework.config.backup.php and then ‘svn revert framework.config.php‘. Then copy your DB_* entries from the top of the backup file into the reverted copy. If you edit the conflict by hand, be sure to ‘svn resolved framework.config.php‘. You can make sure you cleaned everything up with ‘svn status‘.

It’s probably needless to say; but if you don’t use Subversion you can ignore this announcement. ;)

Thanks!

-Jeff@WGM

We’ve implemented a short-term/long-term caching strategy in 4.0 (Build 603) which has significantly boosted performance.

I’ve included a bit of technical information for anyone interested. If this kind of thing puts you to sleep, the above sentence pretty much says it all: with build 603 you should be finding the system even faster than it was already. During the maintenance window last weekend we updated our hosted Cerb4 helpdesks to this version.

Now for the technical details:

Long-term vs. Short-Term Caching

Long-term caching stores frequently-read and infrequently-changed information from the database. The cache is invalidated and re-cached when the source is updated in the database. This long-term cache can be shared between all the helpdesk workers and community tool visitors, and it drastically reduces the load on the database. This caching strategy is especially useful for lists of workers, groups, plugins, mail routing rules or global settings. Any time we have an ID of one of these objects we can simply refer to the cache (to display names, labels and other information) rather than doing expensive joins from the database.

Short-term caching exists for the life of a single page request by a single worker. Often a single request will need the same information (like a list of workers) multiple times when rendering a page. After the initial request is made from the long-term cache, it’s stored in short-term cache so any subsequent requests are lightning fast. Even though the long-term cache is almost always faster than talking to the database, there’s still some overhead involved in talking to it rather than memory. This depends on the caching options from framework.config.php — by default the long-term cache is on the filesystem, though more advanced options like Memcached can be easily enabled.

The Impact on Cerb4

The long-term cache strategy has existed in Cerb4 since the beginning; but we’ve recently realized it could become pretty inefficient on high-load systems as the filesystem became a bottleneck for frequently-requested caches.

This was happening because the long-term cache was retrieved from the disk every time it was requested. For a process like the parser, which is loosely-coupled and modular, some caches are requested from events each time an e-mail message is parsed. If you download 100 messages you have 300+ cache requests. It obviously doesn’t make a lot of sense to be accessing files on the disk 300 times during the same page load. You can imagine the performance-drain effect this has if you have a helpdesk with a lot of concurrent workers all causing frequent hits on the disk cache as well.

The first impulse many programmers have to address this is to “cache the cache” in a local variable. This isn’t as simple as it sounds when you’re dealing with loosely-coupled, object-oriented code. You can’t simply pass references around to methods; and it doesn’t make a lot of sense to implement static patterns like Singleton for every object, or to bloat the API with another specialized global registry just to store cache output.

In prior builds, we were directly exposing the Zend_Cache class (of the Zend Framework) from Devblocks to handle caching. This left short-term caching up to the programmers.

With this build we’ve abstracted the cache manager (DevblocksPlatform::getCacheService()) to automatically handle short-term and long-term caching without requiring any special code. Existing cache code should continue to work exactly as it was written, just much faster under higher loads.

Enjoy!