Alright! Earlier today I promised to share some helpdesk security tips.

If you look at the .htaccess (or .htaccess-dist) file in your /cerb4 directory, you’ll notice the following lines:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . index.php [L]
RewriteRule ^(.*/)?\.svn(/|$) - [F,L]
RewriteRule ^(.*/)?api(/|$) - [F,L]
RewriteRule ^(.*/)?libs(/|$) - [F,L]
RewriteRule ^(.*/)?plugins(/|$) - [F,L]
RewriteRule ^(.*/)?storage(/|$) - [F,L]
RewriteRule ^(.*/)?templates(/|$) - [F,L]
</IfModule>

This file sets up some basic rules, based on URLs and paths, to tell the web server that we don’t want anyone snooping around our private directories. You don’t want shady visitors poking around your /storage directory (which holds incoming e-mail sources and attachments), or your /libs/devblocks/tmp directory (which holds caches that could be used to peek at your helpdesk data or crack your worker logins).

In a perfect world, we’d be creating these directories completely outside the web-accessible path. It’s a fairly easy tweak for those so inclined (who have proper server access); but for the average shared-hosting installation such a setup is often not possible. Our default installation needs to remove as many obstacles as we can without compromising too much. The beauty of Cerb4’s new design is that you can start customizing your installation from our simple, working baseline without losing your ability to easily upgrade.

If your web server isn’t capable of parsing .htaccess files (e.g. they’re disabled, you use IIS, etc) then you need to block the list of paths above using something like “directory security”. Practically every web server will give you this ability, and most control panels in shared hosting environments will as well (e.g. cPanel, Plesk).

Go ahead and test your helpdesk URL to make sure you have something in place to protect these directories. For example, here’s what our online demo shows:
http://demo.cerb4.com/admin/storage
http://demo.cerb4.com/admin/plugins

The absolute worst result you can get is a directory listing that shows all the folders in these directories. If you see that, make sure you get some administrator help immediately (you can send them to this URL).

It’s possible to set up directory security on your entire helpdesk URL (/cerb4/*), but there are a few important things you need to keep in mind if you want to do this:

• You’ll need to allow your community tools to connect to your helpdesk URL. At the moment our reverse proxy script doesn’t have the ability to do HTTP authentication. In an advanced configuration you could allow the IP of your community tool web server to bypass the authentication.
  I’ve gone ahead and opened up a development task to add HTTP authentication awareness to community tools: http://wgmdev.com/jira/browse/CHD-679

• Your cronjobs or scheduled tasks will need to do HTTP authentication to talk to /cerb4/cron or /cerb4/update. This is easier because command-line tools like wget support this already.

• If you use our Web-API plugin, you’ll need to add some extra code to your application to deal with HTTP authentication.

You should be fine with directory security on specific subdirectories without locking down your entire site.

A more practical approach to securing your entire helpdesk path or domain would be to lock it down by IPs rather than passwords. However, this can be very cumbersome — to the point of not being worth doing — if you have a lot of helpdesk staff. You can do this from the web server or from your firewall. You’ll still need to permit your scheduled tasks, Web-API users and community tools (from your public web servers) to connect to the helpdesk.

And while “security through obscurity” is nothing you should critically depend on, some simple common sense can go a long way too:

• Use SSL.

• Rename your /cerb4 subdirectory to something less predictable. If you’re using a subdomain for your private helpdesk (not your public tools), you should try to make it something harder to guess than “helpdesk”.

• Don’t make public links to your helpdesk URL. To the degree possible to avoid, this also includes e-mail, bug reports and forum posts. If Google has indexed your helpdesk URL, then you’re already more immediately vulnerable to every published exploit.

• You could use a non-standard port for your helpdesk web server, if possible. If you control your own server it’s pretty straightforward to set up a separate Apache instance on a port like 9999. The downside is being slightly inconvenient to your helpdesk staff if they have an aversion to browser bookmarks or shortcuts. Get used to people going “the helpdesk disappeared!”.

• You could also host your helpdesk on your office intranet, using a domain like “http://helpdesk/” instead of a web-accessible IP. This won’t impact scheduled tasks that access your URL as long as they run from inside your network. You could permit your public community tools, or Web-API users, to access specific paths of your intranet helpdesk by opening up a non-standard port on your router and filtering traffic for specific URLs from specific IPs.

I wouldn’t really go as far as even calling most of these notes ‘recommendations’. These are simply options available to you.

However, make sure that you can’t access the URLs listed at the top of this post on your helpdesk from your web browser. That should be the very least that you do to lock things down.

A security exploit has been reported that demonstrates the viewing of some helpdesk content without an active session. Please upgrade to 4.0 (Build 600) as soon as possible to correct this important issue.

The flaw has to do with how we authenticate controllers (the first ‘/slash’ commands in the application, like ‘/display’ or ‘/kb’) that aren’t standard helpdesk pages. Pages that you can access from the menu handle security inherently, but functionality that doesn’t tie into the web interface (like ‘/cron’, which is secured by IP addresses) has to handle its own authentication based on the use case.

This preliminary patch addresses the reported issue directly. We’re going to run a full audit through the code to ensure other functionality isn’t susceptible to this same flaw.

If you use Subversion to update Cerberus Helpdesk 4.0 (and you really should be), you can simply issue the console command “svn update -r 600” from your /cerb4 directory on a Unix-based server. If you use Windows, you can use a graphical client like TortoiseSVN to ‘Update to Revision‘ 600 from the right-click menu.

After the code audit, I’ll do a follow-up post this afternoon on the blog with other security considerations. For example, if you wrap your helpdesk URL in HTTP Authentication from the webserver you can add another layer of protection. You’d just need to make sure your cronjob/task (using something like wget) is using the HTTP Authentication. I’ll explain how to do that in my next post.

(We’ll be upgrading all our hosted helpdesks immediately.)

This weekend we’ll be performing some upgrades and maintenance on our Cerb4 hosted helpdesk infrastructure.

We’ll be:

• Adding more RAM to the database machines.

• Adding non-RAID storage to the main hosted machines. This will take a lot of burden off the local disks as we start doing even more frequent backups.

• Migrating some helpdesks to balance load inside the network.

• Upgrading everyone to the latest stable version (currently 4.0 Build 593 602).

The hardware upgrades should be rather quick (minutes if all goes well), and that is the only planned major downtime during this maintenance window.

Moving helpdesks around our network and upgrading them to the latest version will take several hours cumulatively, but each helpdesk should only be impacted for a few minutes. Your hosted helpdesk IP may change but your URL won’t (so ping it and update any firewalls, etc).

Thanks to those of you who are hosting your helpdesk with us! We have a lot of exciting improvements on the way (distributed hosting for large scale helpdesks, cloud-computing/virtualization-ready Cerb4, off-site backup deliveries, and more).

Update 15-May-2008: Our new expected upgrade window is Friday, May 16th from 9:00PM to 10:00PM Pacific Time. This maintenance period should have even less of an impact on Cerb4 hosted helpdesks than planned since we already did the helpdesk upgrades this afternoon with the release of our latest security patch.

Up until a few weeks ago we’ve been doing weekly off-site backups for our hosted helpdesk infrastructure. Local backups were done more frequently, but usually not as frequently as nightly due to performance considerations (and the fact there’s really no such thing as “nightly” to our international customer base — someone is always impacted).

Backups on our scale are a bit more complicated than a single helpdesk, where a simple mysqldump is usually enough to get the job done. We’re hosting helpdesk customers, especially with legacy Cerb2+Cerb3, who have been with us for years. Between those years of e-mail and attachments it’s not uncommon to see 10GB+ databases. That’s a lot of redundant data to be dumping out to *.sql files on the disk — not to mention the redundant cycles wasted on re-compressing it, and wasted bandwidth on transmitting it off-site.

We’ve done a few things to optimize our process over the years:

• We use mysqlhotcopy to copy database content directly (*.frm, *.MYD) and we don’t backup indexes (*.MYI), as they can be fully regenerated during an eventual backup recovery.

• With Cerb2+Cerb3 databases, we also don’t back up the search_index or trigram tables since they can also be regenerated with a ‘re-index’. These inefficient tables (thankfully!) don’t exist in Cerb4.

• When we designed Cerb4 from scratch, we took what we learned about these accumulative Cerb2+Cerb3 inefficiencies and engineered with them in mind. That’s why our attachments are no longer in the database, and why we rely on database indexes for searching again instead of rolling our own indexes inside the database (among hundreds of other design improvements).

• Having these Cerb4 attachments on disk, and named by their unique, incremental database ID, makes it trivial for us to do incremental backups (i.e. only backing up this week’s new attachments and not the entire directory or database table, then aggregating them with the off-site backups).

• We’ve started using Amazon’s Web Services (EC2 + S3) to compress backups off-site. This saves a lot of processing power that otherwise goes to waste compressing backup data, just to delete it the next day (or week) and replace it with fresher data.

We’re still a bit handicapped by these cumbersome databases on the Cerb2+Cerb3 hosting end, but we’re going to stop let it affecting the Cerb4 backup policy. We’re going to start doing daily backups for all Cerb4 helpdesks.

Running backups currently impacts our performance, in a large part, because we’re copying dozens or hundreds of gigabytes from the local RAID to another location on the local RAID (so all disks are involved in jumping back and forth). It would be a lot more logical to do these backups to drives that aren’t involved in serving real-time content to users. This isn’t a revelation, it’s just something we’d sacrificed to economy pricing. Having our new hosting prices based on storage helps us factor these realistic costs in.

Through our optimizations and audits over the past couple weeks (to figure out what’s holding us back from keeping ever-more-frequent backups), we found a major inefficiency on our scale is “original_message.html” attachments. These are the HTML versions of incoming e-mail that usually have a plaintext alternative (which is what we display in the GUI). If an e-mail is HTML-only we end up generating the plaintext part from the HTML.

We’re hosting several hosted helpdesks who have 500,000+ tickets, who never delete a thing (spam or otherwise), and who collect customer through website forms that always generate an unnecessary HTML part. One helpdesk in particular has 530,000 attachments, and ~527,000 of them are a redundant HTML copy of the messages already in the helpdesk.

From now on we’re not going to guarantee that we’ll backup attachments named “original_message.html” — it’s going to be at our discretion. This will save a lot of resources wasted on copying, compressing and transmitting redundant data. It will also shorten the backup windows.

Thanks!

Putting a Price Tag on Bits and Bytes

Software is a tough business, but pricing software is an even tougher business. As obsessive full-time developers we often want as many people as possible to be using our projects, but we also have to strike a balance with the fact there are bills to pay and families to feed.

The price you’re paying for our software has to factor in the costs of ongoing development and support. In our case, the Cerb4 licensing cost is basically just passing around a (slightly-coerced) tip jar for people who like what they see so far so development can continue. As developers we like to constantly be stuck thinking about the future and what new things are possible from the pieces we’ve just built.

Customers tend to approach things differently. Very few prospective users are genuinely thinking about future potential when they see a price tag on software. They take a look through all the corners of a demo and website to get a feel for what is currently possible with the software – probably because they’ve learned over the years that developers can be incredibly undependable in their estimates about new development (and they’re right; but hey, we’re working with ethereal “thought stuff” here).

You’re probably thinking this lead-in sounds like a typical justification for raising prices, right? Well here’s the curve ball… we think we’ve been pricing Cerb4 too high based on its exciting future potential that (for the most part) only we can see right now.

A few months ago we switched over to per-user pricing because it seemed like a great way to offer a lower entry price for companies with few users, while companies with more users could contribute more (based on the theory software seats scale like office chairs). But that’s proven pretty artificial, and for these few months since that decision we’ve been dealing with several tough questions about per-user pricing, like: What if I want to deactivate workers who have left the company without deleting their history? What if we’re a small company that has dozens of volunteer helpdesk workers?

It has recently dawned on us that we’re just setting up an artificial obstacle for people with these worker limits. If you buy into our project, we want you to feel like you own the software. It’s not our style to litter the project with hooks and throttles just for the sake of cash flow. Just like overly-aggressive license validation procedures, it ends up punishing all the paying users out of the paranoia of going broke. The reality is that if we’re doing something genuinely useful we’ll find a way to keep the rent paid, the lights on and the fridge stocked. We may not become the next Microsoft or Salesforce.com with that mindset, but at least we won’t be holding you guys back when you decide to spend a few hundred dollars to use Cerb4 for your mission critical e-mail.

So here’s what we’re going to do: We’re going to remove the per-worker limitations for everyone. Those of you with 5 worker licenses are likely going “Woo hoo!” while those with 25+ worker licenses may be going “What the hell, Jeff?”. Please keep in mind what we’re trying to do here is for the project and community before ourselves, and the fact we may come off as manic-depressive in our search for ideal pricing from time to time has to do with the fact we’re developers, not prescient economists. We’re not fickle, we’re just incrementalists. Each tweak is based on a lot of ongoing observation about the impact of each decision in the real world (which is pretty hard to predict). We’ll continue to find ways to reward people who’s early contributions have helped us grow over the years.

The Impact on Owned Licenses

We’re setting the “owned license” base price for the project to $499 with no artificial limitations. We’ve brought back the small business discount for companies making less than $250,000 USD gross revenues per year which knocks 20% off the price (to $399). Educational institutions get a similar 20% discount (also to $399). Users upgrading from Cerberus Helpdesk 3.x get a 30% discount (to $349, the discount used to be 50% but it was against $995 for unlimited, so you’re still saving about $150 more from this tweak).

The Impact on “As-a-Service” Hosted Licenses

On the hosting “software as a service” end, we’ve stopped tiering pricing by users as well. The scaling issues we face in managing servers have very little correlation to total users, and more to do with how companies manage their e-mail behaviorally (e.g. never deleting spam, receiving lots of work-in-progress attachments). We’ve moved to a much more straightforward pricing model on hosting of just basing it on storage. All hosting plans from today will start with 5GB of storage for $49/mo. 5GB additional storage is +$25/mo and 10GB additional storage (prepaid) is +$42/mo. If this was purely dollars for hosted gigabytes it would be a bit expensive, but this covers the software, upgrades, support, phone support, backups, server monitoring at 4AM, and all those things which you’re glad we’re doing so you don’t have to.

With that Out of the Way…

I can’t guarantee that several months from now I won’t be sitting here telling you something else based on future information and observations; but with all the information and history we have at our disposal right now we feel this is the best direction for funding Cerberus Helpdesk’s ongoing development (so we can continue to reach for all that exciting potential we’re always talking about).

If you have any questions about what we’re doing, how we’re doing it, or what we’re aiming for, ask away! If you’d like to rant, send me the URL to your forum thread and I’ll read through it and post my thoughts without getting defensive or cranky. Nothing is off limits.

Thanks!

-Jeff

Alright! We’ve just done another full round of QA tests against Build 593 to confidently declare it stable. We’ve been using it live on our own helpdesk for about a week.

There are a couple dozens tweaks and improvements included, but our major focus was on bringing knowledgebase management back to the worker GUI, integrating the public knowledgebase with the Support Center (to remove the need for two public tools) and performance improvements.

We’ve also switched over to requiring MySQL 4.1 to vastly simplify fulltext indexes for quicker searches, to get subquery and transaction support, and to move forward with internationalization. It was really useful to gather community feedback about this change ahead of time in the voting booth on the forums.

When we had to shuffle some of our hosted sites around during the server emergency last week, we ended up temporarily putting more sites on a single server than we normally would have liked. It was interesting to observe that about 100 concurrent copies of Cerberus Helpdesk 4.0 ran very usably on a mid-range Debian box (Dual Xeon 3GHz, 2GB RAM, 2×500GB RAID-1). In practice, for that kind of load, we’d throw more memory at the machine and probably bump up the RAID concurrency. The hard disks are almost always the bottleneck if you can’t keep databases entirely in memory.

Naturally we’re not in the habit of overselling a machine’s resources, but it gave us the opportunity to look at bottlenecks with that many copies of Cerb4 running. We used this information to add some last minute performance improvements to Build 593. One of the most effective was improved database indexing on the message_header table – which is used to thread incoming messages with the messages they’re replying to, as well as to display the original headers while reading ticket messages. There were a couple other missing indexes, and a few places in the code we could reduce unnecessary queries.

For the knowledgebase, you’re now able to use nested categories again (as was heavily requested). The top-level categories of the knowledgebase are now called “topics”, and you use topics to decide what branches of the knowledgebase to display on your various public community tool instances. You can display more than one topic for a particular website. We’ve found it’s most effective to use your products and services as the basis of topics.

To take advantage of the new community tool changes you’ll need to go into ‘Helpdesk Setup->Community Tools’ and choose your knowledgebase topics to display for each tool.

You can find the full changelog for Build 593 in the forums.

Around 1:30PM Pacific on Monday (April 28th 2008) we had one of our Cerb4 Hosted machines start writing garbage data all over its RAID drives. This corruption turned MySQL database files into directories from random fragments of the disk (website files, etc). It was a complete mess.

We lost SSH access to the box around that time and initiated a remote reboot. When the machine came back up the primary partition was mounted read-only (which pretty much always means no good).

A full file system check is needed, but we decided to take the proactive step of moving all hosted sites from that machine to a standby server rather than letting the corruption get worse.

Our off-site backups didn’t contain corruption, though a few of the on-server backups did. We do our weekly off-site backups early on Sunday mornings (when the traffic is lowest). In the worst cases, we had full databases for any site that’s only missing Monday morning.

As soon as we realized what was going on, we shut down the scheduled tasks on that machine so we wouldn’t be losing your e-mail to a corrupt environment. This buffered up mail in your POP3 accounts. This is our standard procedure, which gives you the ability to access your critical e-mail directly from your POP3 mailbox (with a “leave a copy of mail on server” client).

The way Cerb4 works, the corrupt database tables also caused the helpdesk parsers to lock (as designed) and started buffering up downloaded e-mail to the disk. This, combined with turning off scheduled tasks, should have prevented mail loss during the day.

We took a little longer than we would have liked in restoring some helpdesks this afternoon because we wanted to maximize the amount of recent data we were recovering, instead of just falling back to the backups when it wasn’t always necessary (and would lose recoverable data). For some sites, the only corruption was on an infrequently changed table like ‘worker’ which we could restore independently. The majority of sites on that machine had no corruption at all (and the busiest sites were affected the most).

We currently have all sites and databases restored on a fresh server. We’re importing the /storage directories which contain attachments and unparsed e-mail. These imports won’t affect your ability to use your helpdesk, but you may not be able to open some older attachments until the process is done. We’re establishing another standby server.

After things settle down a bit, we’ll reflect on this 12 hour marathon and think about what we can do to get sites back in action quicker from a total machine failure. The off-site backups will continue to play a big role. Cerb4 is “componentized” enough that our hosted sites are incredibly portable on our network; but we also need to have our standby machines already mirroring the DNS for a seamless changeover.

If you’re still having trouble accessing your hosted site it’s likely just a DNS caching issue from the server migration. Make sure your *.cerb4.com site is resolving to the IP “72.232.216.42”.

Thanks to everyone for your patience during your calls, e-mails and livehelp chats today. This isn’t a situation we’re going to have to go through very often – in fact, it was a complete nightmare — but it’s at least slightly comforting to know our off-site backup process gave us day old data as our worst case option.

[http://www.cerb4.com/forums/showthread.php?t=874]

[Alternate URL: http://forum.cerb4.com/showthread.php?t=874]

Since 4.0 development has been moving so quickly, we’ve just had everyone pulling the absolute latest source code from Subversion (technically called the “HEAD” revision). This was a good thing during early development because it kept us all on the same page.

However, with 4.0 no longer in beta, we really need to improve this process so people know they’re running the latest stable code and not something we worked on 5 minutes ago, or bleary-eyed at 3AM. Most of the time we’re pretty thorough with our local tests, but they’re no substitute for the acceptance tests and browser-compliance tests Joe@WGM runs in Q/A. You guys have access to our main repository, so there’s no guarantee the absolute latest code has gone through Q/A.

Our philosophy with 4.0 has been to *not* make a big deal about point releases in an attempt to discourage ourselves from batching up functionality and fixes to make a “marketable release”. We want to give you real-time access to the fixes and features that you care about. Those of you who need a specific feature (or want to help test) can upgrade 5 minutes after we commit changes. Everyone else can be confident we’ve fully tested the incremental upgrade they’re going through.

It’s not that we haven’t been testing things carefully all along, but we haven’t been very good about telling you after we run all our tests.

So here’s what we’re going to do:

• Every other Monday we’re going to make a new milestone build that will go through Joe@WGM’s entire acceptance and browser test suite. Anything that fails will go back to development as a top priority. This is pretty much how things work now anyway, but it leads into:
• Once the acceptance tests all pass, we’ll update the latest “stable” build on our on-line demos, instant evaluations, hosted helpdesks and the downloadable ZIP file.
• If you run Cerb4 on your own server, you can upgrade by simply issuing the command “svn -r nnn update” from your /cerb4 directory at the server console (where ‘nnn’ is the version number posted on the site). In Windows you can use TortoiseSVN to “Update to Revision” by right clicking the cerb4 folder in Explorer.

Thanks for being enthusiastic enough about the project that this is even an issue!

First off, thanks for supporting Cerberus Helpdesk development by previously purchasing a version 3.0 license! Contributions like yours have allowed us to push forward full-time for over six years.

Over those six years, the project has gone through some big changes while striving to stay agile for a growing community. The early community was mostly made up of techies; but over the past few years we’ve had increasing interest from people in all industries who are still taking the plunge to web-based e-mail from POP3/IMAP mailboxes and desktop e-mail clients.

We’ve never been the type of project to increase major versions for marketing reasons. We increment our major version when we feel the project and its concepts are going through a fundamental metamorphosis that, despite increasing version numbers, doesn’t require us to stay trapped by a compatibility with the inefficiencies and missteps of the past.

The latest metamorphosis has been version 4.0, which is our answer to the question “What would you do differently if you could start over with what you know today?”.

At first glance you’ll probably notice things are different (and slimmer). Please don’t let this deter you, as it’s part of the process. 4.0 places its major emphasis on customization, which means plug-ins can add or change functionality based on the needs of specific groups. It means you aren’t boxed in by a “one-size-fits-all” mentality. This clean slate allows you to add, build or share the pieces you need without having to deal with the bloat of the pieces you don’t.

For example, at WebGroup Media we’ve added custom tools to our helpdesk to manage our forums, CRM opportunities and product licensing. We’re able to quickly know more about our specific customers when looking at a ticket without cluttering the project for everyone else. A big part of 4.0 development is going to be building and fostering the development of similar tools for various groups. For this reason, we want to keep the core (shared) project as slim as possible.

If you feel your past investment in our project has been worthwhile, please consider upgrading to 4.0 and supporting this effort. As a 3.0 license owner you’ll receive a 50% discount. Don’t wait around for the perfect project to suddenly materialize without your input, stop by the forums and tell us what your ideal helpdesk should do.

We’ve been asking you guys what you feel about the new $50 per worker pricing ($25 per worker on upgrades) since we made the switch. The change to per worker pricing has freed us from having to hold functionality hostage, since we really feel every feature is helpful to organizations of all sizes (heck, I use most of them on my own webmail!) So far the reception has been positive, but with a few lingering issues to work out.

The main remaining contention is about where the cutoff should be for unlimited workers. Many people feel that the cutoff being 50 workers is too high. It’s true that the average number of workers on licenses since we’ve been asking has been quite a bit less than 50. We can also sympathize with organizations that have a lot of community volunteer or part-time workers.

Looking at the data from recent orders, we feel a better cutoff for an unlimited license is probably around 20 workers. That still gives a good discount to smaller teams while keeping the price reasonable for larger teams. Unlimited workers after 20, coupled with a small business or upgrade discount, reduces the unlimited worker license by up to 60%.

We’ve decided to trial offering the unlimited license at 20 workers through April 15^th 2008. The 50% discount for 2.0 and 3.0 license upgrades still applies as well. That’s $995 for unlimited workers on new purchases or $497 for an upgrade from 3.0. Both prices include all version 4.0 updates.

If you already purchased or upgraded to 4.0 and paid for more than 20 workers, contact us and we’ll upgrade you to an unlimited license at no cost. If you paid for less than 20 workers, we’d be happy to upgrade you to an unlimited license using your initial purchase price as a discount.

Thanks for the constant stream of candid feedback!