We have some major improvements with the Address Book and Custom Fields ready for the next update (which is going through Q/A now).  The Address Book’s potential had been neglected up to this point, and we’re really excited about the new possibilities.

Below is a quick screencast that I put together to show the improvements in action.  This includes:

• Modeling a brand new workflow in the Address Book with a few custom fields.
• Using custom fields as new list columns and search filters.
• Leaving notes on Organization records to track progress and contacts.
• Exporting Address Book information to CSV or XML using the built-in search functionality.
• Using Bulk Update on Organizations and Addresses to save monotonous data entry.

I realized after I finished the video that I didn’t include an example of setting custom fields using the “peek” pop-ups; but you’re able to quickly enter any of that info where you normally use peek (lists, search results, record display).

Here’s the video (it runs about 9 minutes):

We hope to have this release available as a stable update by the end of the month.

Think we’re on the right track?  Are we missing something?  We’d love to hear some feedback in the comments.

-Jeff@WGM

I know a lot of my conclusions here are going to be forehead-slapping obvious to some of you — but I’m going to post this anyway!

One of the big things people have been asking for in Cerb4 is having custom fields on contacts (organizations and addresses).  You could add custom fields to those records in Cerb3, but the approach left a lot to be desired on performance and code maintenance/simplicity.  I’ve always felt like the custom field and search results combination was the most complicated part of the app’s design.

I know I’d been groaning and avoiding custom fields growing beyond tickets because of the cruft introduced by earlier versions.  But when I finally sat down and thought about it, I came up with something as elegant and speedy as the rest of Cerb4, which allowed us to add custom fields to every type of object in the helpdesk.  This is one of those “How did we ever live without this!” moments for me.

Once I was over my bias about custom fields I started to realize where they led:

• One of the worst parts of introducing new features is having to decide on a neutral design that tries not to do too little or too much.  Inherently, that makes it really hard to please anybody.  With custom fields we can provide a new type of object, like a “call” or time tracking entry, without having to feel like any fields we come up with will box people in to our particular way of working.  This solves a few standing, nagging issues — like how people are expected to keep track of which time tracking entries are billed (through an outside invoicing process).
• Some existing records in Cerb4 now have fields we feel are superfluous.  We can now use the upgrade step on a new release to move these fields into custom fields, and people who no longer want them can remove them entirely.  A good example of this is the “Account #” field on organization records — if you’re not using it, it’s just wasting space by being there.  It confuses teams who don’t have a pre-existing account number for their customers.  The same is true of the “Service Level” feature, which will probably be pushed to custom fields very soon to make way for better SLA functionality.
• It’s tough to get people to agree on how granular a field like “priority” or “status” on tickets should be.  Those concepts are useful to a lot of people, but we don’t feel like they warrant being required.
• Sometimes different teams want to assign objects to workers (e.g. organization records to account managers) that we didn’t build into the system.  We really don’t want to add the concept of assignment to things unless people really want the workflow — otherwise it’s just another unused field taking up space.  By letting people add custom fields to those objects to track ad-hoc assignment, everybody wins.
• This moves dozens of other great ideas forward by removing the fear of bloating Cerb4: assigning products to organizations in the CRM plugin, the call logging plugin, SLAs, etc.

To risk getting a little technical, the revelation to me isn’t so much about what custom fields can do — that much was obvious — but I’m excited about having designed custom fields as a “platform service”; which is a fancy way of saying Devblocks (Cerb4’s engine) now makes it painless for developers to allow custom fields everywhere.  That even includes objects that come from third-party plugins.  For example, if someone out there created a live chat plugin, they could add complete custom field functionlity to transcripts (list columns, search filters, export, peek data-entry) in a matter of minutes; and it would work exactly like the rest of Cerb4.

You can expect to see a lot of interesting things come out of this in the very near future!  The mental gears are spinning at full speed again.

-Jeff@WGM

Twice this week (Jan 14th + Jan 16th), about 5% of our On-Demand users likely experienced a brief outage during our night shift (Los Angeles, GMT-8).

Specifically, these users were able to pull up their helpdesk by URL but they couldn’t log in and were given an error message about connecting to their database.

Two nights ago when the issue first happened, we believed it may have been an isolated coincidence of several users purging massive amounts of spam at the same time; synchronized due to some instructions we sent out a few hours before.  It seemed plausible, considering the issue came out of nowhere after months of smooth sailing.  That situation probably still played a part, but it wasn’t the whole story.

This morning we had several more reports of the exact same issue; but this time we managed to catch it in action. The affected database server had no limit on connections by an individual user, and a single helpdesk was hording all the available slots by creating thousands of connections per second through their public Support Center.  A classic “denial of service”. However, there’s no evidence it was malicious.

From time to time, a user will benchmark our servers during an evaluation.  With reasonable settings everything will be served up just fine.  With excessive settings, simulating thousands of simultaneous connects for a helpdesk that’s going to have 5 users, we’re supposed to be throttling that account from affecting the entire server.  For this particular set of machines any client was able to use all available connections.  For the past year or so we’ve apparently just had some very polite customers on that hardware.

The trick with throttling is to set it to where it will never be hit by normal peak usage, and only by some runaway process.  We’ve re-established a per-user connection limit that’s much higher than our average usage statistics show across our network, but still low enough to not allow an individual helpdesk to monopolize the entire database server.  The potential still exists for a concentrated attack to distribute across multiple sites, but we realistically have to build in failure somewhere.  We could raise the maximum connection limit so high that nobody is ever refused, but servers would still become so overloaded they’d cease responding anyway.  By failing at a reasonable point, the machines are still fully usable to find and correct the source of an attack.

Luckily, in this situation it was just a simple misconfiguration issue.  We aren’t having chronic performance problems, and once an issue like this occurs it’s not unexpected for it to rear its head a few times in quick succession before it’s caught.

I could do an entire blog post on the economics of high-availability, and I may go ahead and do that today while it’s on my mind.  The basic idea is that it costs at least 200-300% more to eliminate the last 4.5 hours of downtime in 99.95% availability than it costs to be up the other ~8,762 hours.  It requires duplicate hardware waiting for its 22.5 minutes of glory, on average, per month.  We’ve gone back and forth about cost vs. excessive reliability many times on our end, but we still feel you guys would rather pay a cheaper rate for 99.95% availability than paying several times as much to approach 99.999%.

All that said, our minimal downtime by design should be planned and at off-peak times. In this case it wasn’t planned or off-peak for a portion of our users, and was due to a misconfiguration on our end. We apologize for that.

We’ll do what it takes to prove we stand behind our service.  Feel free to leave comments with questions or thoughts.  You can also contact me directly: jeff (AT) webgroupmedia.com

Hopefully on the next few posts I’ll be writing under better circumstances! :)

-Jeff@WGM

You can add Cerb4 to your Dock and Applications folder in Mac OS X using Fluid.app.  This makes it really easy to switch between your helpdesk and other windows while you’re working.  You’ll no longer have to hassle with losing your helpdesk in a sea of browser tabs and windows.

This takes about 2 minutes to set up by following these instructions on the wiki.

Enjoy!
-Jeff@WGM

We just added a new page to the wiki about using Cerb4 and Gmail together.

Cerb4 and Gmail aren’t mutually exclusive; and there are actually a lot of reasons why you might want to run all your mail through Gmail first, even when you read and reply entirely in Cerb4.  Here are a few good reasons for configuring your helpdesk this way:

• Gmail can act as a free, real-time backup of all your incoming and outgoing helpdesk e-mail.
• If your helpdesk is inaccessible for any reason (e.g. server network outage) you can still read and reply to e-mail.
• You’ll get an extra line of defense against spam, and reduce the amount of junk mail your helpdesk has to download and parse.
• You can use Gmail as your incoming (POP3) and outgoing (SMTP) mail server which allows you to scale Cerb4 easier, or run your helpdesk on a virtual server.  It also reduces the number of things you have to manage.

Here’s the full article: http://wiki.cerb4.com/wiki/Cerb4_and_Gmail

-Jeff@WGM

We had about a dozen reports about database connectivity issues this morning from Cerb4 On-Demand customers that all had a particular database server in common. The issue resolved itself last night and didn’t trigger any logs or network monitoring alerts (e.g. uptime, load). There wasn’t any data loss.

Our best theory at the moment is that by freak coincidence a bunch of customers followed the instructions we sent out last night to purge their spam histories at exactly the same time (we’re happy people found the instructions useful!).  The timing makes sense.  And even though we leave plenty of extra capacity on all our machines, it’s possible that this database server (MySQL) was “blocking” while waiting on the disk, which would have queued up connections until it hit a limit that stopped accepting new requests.  That’s entirely consistent with what people reported.

If that’s what happened then this shouldn’t be an issue that pops up very often.  The database has to do a bit of work to remove 100,000s of spam tickets at the same time when people aren’t in the habit of cleaning up more often.  That’s the exact reason we felt it was important to send out the best practices guide for Cerb4 anti-spam.  The unintended consequence was kind of a digital “bank run” to clean up databases.

While overloaded, mail just wasn’t being checked — it wasn’t lost.  Once the maintenance finished mail was caught up.

We’ll keep an extra eye on that particular hardware just to be on the safe side.  I could throw around some stats about our uptime 99.99% of the time, but those of you who were impacted by this know what our service and performance has been like up to this point. :)

If you have any questions about this, or anything, feel free to comment or open up a ticket from our website.

Thanks!
-Jeff@WGM

Configuring groups to quarantine probable spam.

Cerb4’s anti-spam system is a huge saver of both time and sanity, but it has the potential to be used inefficiently if it’s used “as is” once your helpdesk is running.  Here is an overview of how Cerb4 deals with spam, along with tips based on our own workflow (at WebGroup Media) to get you started:

http://wiki.cerb4.com/wiki/Best_Practices:Dealing_with_Spam

Enjoy!
-Jeff@WGM

In January 2002 — at the tail-end of the last big economic shakeup — WebGroup Media LLC (WGM) had been a full-time venture for about nine months. Ben Halsted and I were the early founders, and we had enough contract gigs behind us to know we really wanted to work on something with a passionate, long-term mission. We were tired of working on projects where the specs called for technology decisions based on marketing bullet-points; leaving us to always learn the better tools in our spare time.

We were still paying the bills back then with a couple contract jobs and a few hundred customers on budget hosting plans. But we had our senses primed for finding the problem we’d enjoy throwing ourselves at for a living.

We had to spread ourselves pretty thin to bootstrap the company and keep the lights on, and accepting all that discounted work kept us in a mode of perpetual maintenance just answering questions and fixing problems. I remember feeling like we’d never find the time to build something new because the workload would reset itself each night.

We’d tried to organize the e-mail workload with several different apps, but primarily we had been using RT. The interface worked just fine, but even with our light volume it started to become unusable pretty quickly. It’s still around, and I’m sure it’s made a lot of progress since then, but at the time we were rapidly exhausting our supply of small talk for passing the several minutes it took to run searches. Beyond that, it’s written in Perl, and most web hosts were already strongly favoring the LAMP stack (Linux, Apache, MySQL, PHP). Pundits say the “P” stands for PHP, Python, or Perl. In the hosting market, it stands for PHP.

January 9th 2002 was the day Ben and I had an IM conversation where we brainstormed about how to break the monotonous cycle and start working on something more interesting. In good spirits, and youthful defiance, we kept joking about how “support is hell”. It was around the same time that we were establishing the mantra we’ve had to this day of “Most Important Thing First!” (or “MITF!”). It didn’t take long to realize other people were probably as frustrated with their shared e-mail as we were. Our biggest problem was the thing we had the most to say about. We had our project idea.

We quickly started thinking of names for the project. Some serious, most not. I’ve long since lost the original logs, but I believe it was Ben who (perhaps jokingly, but now immortally) asked what the name of the guard dog to hell in mythology was. I said “Cerberus”. He said something along the lines of “We should just call it that”. We laughed. It had been a long chat, and we decided to go with the code name of “Cerberus” until something better came along.

Two months later we sold the first copy. The logo still said “Cerberus Helpdesk”. There was no going back!

It’s been a really interesting seven years.

In a lot of ways, we underestimated what we were getting ourselves into (“I know! Let’s make a startup that stores years worth of e-mail and file attachments for thousands of companies worldwide; guaranteeing our office never closes and we never sleep!”). What we definitely didn’t underestimate was how many people shared our frustration with shared e-mail. We’ve had the good fortune of coming to the attention of well over twenty thousand teams in the past few years. All that feedback and positive energy has snowballed the project into something so much better than we had originally imagined. As cliché as that sounds, it’s still true.

I think the big reason this project has been energized for seven years is that it’s still as exciting and relevant today as it was in the beginning. We owe a lot of that to a project culture that’s still willing to take risks and make big changes while pursuing solutions that are as simple as they are powerful. That hasn’t been without difficulty. The transition from Cerb2 to Cerb3 was seen by many in the community as a disaster of Frankenstein proportions (I like to call it the “co-mutiny”); and it’s taken over 2 years for Cerb4, free of the baggage and clutter, to win people over based on its own merits as something entirely new.

Cerb4 is new, but it has seven years and thousands of brains behind it. Seeing that work pay off as people put it to work is hugely rewarding. Even this morning, somebody on the team sent me the link to a forum post where a community member said, “Cerb4 is naked compared to Cerb3!”, quickly followed up with another post by the same person saying, “I have to say, even naked, it’s very powerful and very easy to use. Thumbs up.”

That’s what gets me out of bed in the morning… 2,556 days later. ;)

With the mushy nostalgia out of the way, here’s an entertaining look at the project’s evolution.

The interface today still proudly shows its lineage, but a refined usability has replaced the gaudy attention-seeking of yesteryear:

The project website has gone through a bit of an identity crisis: from a blip (2002), to taking itself too seriously (2005-2007), to defying convention (2008), and settling into a matured self-confidence for 2009.

The project web site history was pulled from the Internet Archive’s “WayBack Machine”:
http://web.archive.org/web/*/http://www.cerberusweb.com

Happy Birthday, Cerberus! And thank *YOU* for being a part of the project!

-Jeff Standen, Chief of R&D, WebGroup Media LLC.
on behalf of the current Cerberus Helpdesk team: myself, Darren Sugita, Dan Hildebrandt, Joe Geck, and Mike Fogg; and retired team members: Ben Halsted, Jeremy Johnstone, Brenan Cavish, Jerry Kanoholani, and Trent Ramseyer; and many more community heroes.