Based on the twenty seven comments I received on this blog entry I couldn't help myself from thinking how much Domino shops are very much like Exchange 5.5/2003 shops in the fact that they don't migrate as quickly as we think they do. While this sampling only represents a fraction of the number of Domino servers out there I can't help but think that broader surveys may yield the same results.
This leads me to the question - "Why aren't companies upgrading their Domino servers to newer versions?". We have in place upgrade capabilities, a proven track record with newer versions of the Domino servers (7.x, 8x) and a large number of qualified Domino Administrators out there. So what can be done to get older versions upgraded? Applications are backwards compatible and should and actually do work just fine. This would bring more revenue to VAR's, ISV's and developers. Or are they not upgrading because they plan on upgrading to something else?
We in the Lotus world often cite the relatively low adoption rates of new Exchange Server versions. Domino 6.5 came out in 2003 and Exchange 2003 also came out that year as well. While I do not know the number of Exchange 5.5 systems still out there (Exchange 5.5 came out in 1997) nor do I know how many shops have upgraded to Exchange Server 2003, I have to estimate that most Microsoft shops are now running at least 2003 and a much smaller percentage running 2007.
[flame retardant removed]
Again the purpose of this discussion is to figure out what "we" can do to get customers off older versions of Domino. Put this thought in the back of your head - "Developing applications for Domino 8x and 8.5x won't run on these older servers".
And for those who might be asking IdeaJam does run on Domino 6.5x through Domino 8.0.1.
Comment posted by Steve Breitenbach04/11/2008 10:24:05 AM
Homepage: http://sclaffer.blogspot.com
Bruce,
As for me, a customer, it comes down to finding the time to do the upgrade. We only have two Domino servers and only 100 or so users, but our IT staff is two people. Finding the time to do the upgrade is a tough thing to do, whether I do it myself or bring in a consultant to do it (usually on weekends for upgrades = $$$ for the consultant.) I would love to upgrade to 8.0.1 right now, but can't find the time to do it right now. It's something I have to plan for in the future (3rd or 4th quarter... sooner if I can get around to it.)
At the same time, I just purchased Quickr licenses and have downloaded Quckr 8.1, but we still need to get a server to install it on (as well as purchasing another Domino license.) Again, that'll probably be around the 3rd or 4th quarter of this year.
Steve
Comment posted by Bruce04/11/2008 10:26:56 AM
@Steve,
Great feedback. Let me know if you need resources for the upgrades as I know lots of affordable Domino talent out there who I am sure would love to assist you with the upgrade.
Comment posted by Gregg Eldred04/11/2008 10:27:28 AM
Homepage: http://www.ns-tech.com/blog/geldred.nsf
In the Domino world, I would split this question into two:
1) Domino messaging servers upgraded
2) Domino application servers upgraded
Looking at things from my little corner of the world, messaging servers get upgraded more quickly than the app servers. When it comes to the app servers, many is the time where I hear "we don't have the budget to upgrade them," or "we can't move because the version of <insert 3rd party app here> doesn't run in R7/R8." Or, "we don't have time to test everything that is running on that server." So, they stay at R5/R6 while the messaging servers are upgraded to R7 or R8.
There are too many variables to simply create an Application Server Upgrader's Redbook. But a wiki might work. 
Comment posted by Steve McDonagh04/11/2008 10:28:36 AM
Homepage: http://dominoyesmaybe.blogspot.com
Bruce
To be honest, and I can only speak for my own experience here, there are 3 main things that hold back on upgrades
1. Licensing - IBM keep moving the f'ing goal posts and for the 500-1000+ seat sites that can be a pain and it has to be said a bit expensive (but no more than any other comparable product). So CFO's are inclined to let maintenance to let maintenance slip and then it's all the pain of "buy-in" when something new comes along and that all adds time and pain to the equation.
2. A general belief that one should not fix what is not broke. This is reinforced by the less than luke-warm reception some users gave previous releases of the client a V5 and V6. It does hold you back when all your work as an admin is dissed as "its still not as nice as xxxxxxxxx". This has been sort of knocked on the head by V8 which is does bring OOOO and AHHH gasps from the great unwashed.
3. Hardware not keeping up with software. In these times of ever tightening budgets the $$'s aren't there for Joe Public on the shop floor to get a new machine, or to go out and buy 1gb of memory for 60 machines just to get the new software to work. As a result a lot of shops sit on upgrades until their hardware catches up with the software.
Reasons 1+3 are prob not true for Finance or IT based business but for the Manufacturing sector where IT is pure "overhead" this fact is reflected in the budget allocations passed down from CFO's and they are REAL buggers to get around
:) See no flames at all..well not from me anyway.
Steve
Comment posted by Corey04/11/2008 10:38:13 AM
Homepage: http://www.conxsys.com/blog
No flame here, but I also don't think that you are looking at this correctly. You are comparing Exchange 5.5/2003 to Domino releases which have very different timelines. You stated that Exchange 5.5 came out in 1997, that would be about the same time as R4.5 (actually 1996, but close enough). So, between 1997 and 2003 there were zero new versions of Exchange released. The Domino community got R5 (1999), R6 (2002), and R6.5 (2003). You stated you don't know how many Exchange 5.5 installations there are, but surely there are still quite a few. How many R4.5 installations are there?
The point is, we do have a higher upgrade rate other wise you would have seen more R4.5 or R5 responses to your question. Instead, I saw a good number of 7.x installs and many comments of plans to move to 8 by end of year. The fact is that we in the Domino community get such regular upgrades from Lotus that many organizations are very comfortable just skipping a version because a new one will be out in a year or two.
Comment posted by Dave Leigh04/11/2008 11:08:35 AM
Homepage: http://www.cratchit.org
Simple. Because the older version works fine.
If a product meets the customers needs, and those needs haven't changed, there is no rationale for the upgrade. Some customers change their needs through continuous improvement. Some change their needs doe to external pressure. But that's only a fractional part of the total pool of customers. Another percentage WOULD change if they understood the additional capabilities of the new system and could think of ways to leverage them. Marketing (better than the marketing that Lotus has traditionally done) would convert these. What remains is a significantly sized group of customers that, due to price or lack of need, simply won't upgrade. Period.
Comment posted by Nathan T. Freeman04/11/2008 11:18:08 AM
Homepage: http://nathan.lotus911.com
I'm with Corey here. I'm confused by your conclusions. Are you suggesting that the penetration of Domino 7/8 upgrades from 6.5 is similar to the penetration of Exchange 2007 upgrades from 2003? I honestly have no idea what the ratio of 2007/2003 servers is, but from your informal request, it looks like 6.x represents, worst case, 35%.
Comment posted by Wild Bill04/11/2008 12:56:17 PM
Homepage: http://www.billbuchan.com
In terms of the medium and large customers I see, there's real reluctance to upgrade (why risk it) and of course real time+money impact. For instance, one 14k user shop I know only has THREE admins (they need about 10).
They fully understand that moving to 8.x is a good idea, but just dont have the resources. As Domino is pretty stable, there's just not the enthusiasm to drive out the new stuff.
If the domino server had an auto-patch or upgrade option (where the server itself would shut down, download the necessary stuff, and do the upgrade during weekend downtime), this might help.. The admins would need some sort of configuration control so they could say - 'Ah 8.0.1 FP2 is out - do that over the weekend' or whatever...
Another customer - 90k users - planned to move to Exchange 2 years ago, and have now managed 2k users. Of course they let their passport lapse, so their servers are all on 6.5x. Now they're trying to move their apps, and to be quite honest, could have web service enabled the lot of them and left them on Domino (but reskinned, etc). Politically, of course, they dont want to spend more money on Domino - huge loss of face, etc.
And the last reason? We no longer have dedicated Domino admins in small to medium shops, and with the same team supporting half a dozen platforms (and of course the lack of mainstream advertising) means that the 'Domino is dead' marketing message from MS has stuck, and no one gives a crap about Domino anymore..
So somehow convincing customers in the second and third examples that in fact it *is* an IBM product with a future - something that I think has worked in the US, but has still to percolate fully across the Atlantic - would really help.
---* Bill
Comment posted by Scott Stewart04/11/2008 01:18:24 PM
It would really help a lot of R8 server upgrades if RIM would have been a little more proactive in releasing a supported-for-Domino 8 BES software upgrade earlier than the current planned release date of April 15, 2008. (At this point, I'll believe it when I can actually download and install it)
We were ready to upgrade as soon as 8.0.1 hit, but these repeated delays by RIM has likely pushed the Domino 8.x upgrades to late 2Q-early 3Q.
I would hope IBM would engage RIM on future releases much earlier in the process and insist that they at least support Domino releases with a Service Pack within days of the gold release. (I know, RIM will likely do whatever they want, but a little "incentive", err encouragement from a large vendor never hurts)
Comment posted by Julian Robichaux04/11/2008 02:12:56 PM
Homepage: http://www.nsftools.com
I'll start by saying that I don't work with nearly as many different companies as some of these other guys, although I have been in a few really big ones. I also have a funny habit of looking at the mail headers when people from Notes shops send me e-mail, which tells you what client version they're on. 
I still see a lot of 6.5 clients out there, although I think the server versions are much more of a mixed bag (lots of R7 servers). I think there are 2 big reasons that people wouldn't have upgraded their clients to 7. (1) There weren't many compelling reasons to upgrade from 6.5 to 7.0 at the client level from the new feature standpoint, although the servers had a lot of good new features. (2) It's a lot easier to test and deploy a [relatively] few new servers versus a [relatively] huge number of new clients.
I also think that ND8 does have some compelling reasons to upgrade at the client level, which will necessarily get people to upgrade servers at the same time. The trick is planning the upgrade -- if you have hundreds, thousands, or especially tens of thousands of users, it's difficult to upgrade anything at the desktop level. Or, it seems like it would be anyway. I'm not an admin...
Comment posted by Colin Williams04/11/2008 03:52:14 PM
Plain and simple: Time. Its just me running 7 servers and 700 users. I spend at most 20% of my time on Notes (admin&dev). I'd love to specialise in Notes at my current employer but they can't see the point in a full time Notes person.
Comment posted by Rob McDonagh04/11/2008 05:55:01 PM
Homepage: http://www.CaptainOblivious.com
Auditors. Plain and simple. The IT departments in publicly traded corporations (enterprises, in other words - prime Domino customers) are all liable to be audited for SOX compliance. Every upgrade requires a detailed test plan, a project plan, and user sign-off.
We can say all we like that this is simply "best practices" being enforced, but the reality is that nobody needed to do any of those things to upgrade a Domino server before SOX hit. Domino upgrades are EASY. In the 4.x-5 days, I'd do the upgrades whenever it was appropriate, and as long as I had backups I was all set. Now? I need a compelling business reason, I need a budget, and I need enough documentation to cover my butt.
That's why my app servers run the versions they do - 6.5x, 7.x, and 8.x - it depends on what was current when they were installed. If it was entirely up to me, they'd all be running 8.01. We don't get to do upgrades because of new features (the carrot) anymore, we do them because of the stick - when IBM says the software is not supported as of X date, we're allowed to budget a massive project to upgrade. Which is absolutely inane, because the actual upgrade will take about 30 minutes, but the prep work will take months and cost much more than the hardware and software combined.
Is the same thing affecting Exchange shops? I would expect so, yes. Since Exchange upgrades (5.5 to 2000 or 2k3, with AD of course, in particular) are NOT easy, though, I think the impact is much smaller. Those upgrades are slow because they're difficult, and adding SOX compliance onto the top of that doesn't change the overall picture very much. They were slow, they're still slow.
Comment posted by Henning Heinz04/12/2008 02:10:02 AM
The little customers I have run older SAP releases (4.6), older Office Versions (XP/2003) and Windows XP/2000, Oracle 8/9 or SQL Srv 2000 (list continued). It is tough to convince companies that have stretched their life cycles to 5 years and more (even for core products) and adapted their IT workforce accordingly. And rolling out two GUI versions of a client does not help either. So what can I do about it? That is a tough question because I think they do it right. I am sorry to say but IBM has to do the work. Upgraded and enhanced templates would be a good sell, reduced client ressources and finally a much needed comeback of RAD. Domino has lost most if not all of its power in this area.
Traveller and DWA Light are already available and very helpful. Maybe Quickr personal too (although I haven't looked at it). This stuff is much easier to sell than e.g. the web services story of Domino 7.
Comment posted by Nathan T. Freeman04/12/2008 04:34:06 AM
Homepage: http://nathan.lotus911.com
@Rob Given that a SOX audit is a verification that you followed the controls that your company defines, why not just define your controls for Domino to make it simple? Or at the least, write a boilerplate upgrade plan that you can easily duplicate.
Comment posted by Bruce04/12/2008 07:11:12 AM
So now does anybody think that "Software As A Service" may start making some sense?
Comment posted by Rob McDonagh04/12/2008 07:12:24 AM
Homepage: http://www.CaptainOblivious.com
@Nathan - I know as well as you do that SOX itself says nothing specific about test plans and project plans and user acceptance sign-offs. But the geniuses inside the SEC have interpreted it in such a way as to turn the big auditing companies into dictators. We don't get to tell them what our policies are - they tell us what an acceptable policy would be. And that acceptable policy, on an app server, involves a full suite of regression testing, with documented test cases and results and a formal user sign-off. And that basically means that upgrading costs way too much money unless it is absolutely mandatory.
Now, my favorite bit about our auditing overlords is their attitude towards OS patches from M$: they must be applied immediately, regardless of possible application impacts, and they don't require any testing - isn't that brilliant? And yes, they've broken our apps on more than one occasion - doesn't matter. 
Comment posted by B04/12/2008 12:37:32 PM
My company was just 500 employees, but we got bought recently by a company with 20,000 employees worldwide. And let me just say, our IT staff doesn't get anything worthwhile done these days. Our users are frustrated, and yeah, the IT staff is too. We laughed at the last team meeting because all we're doing is SOX or audit prep work. We don't do real work anymore. I'm sure no one is happier than our competitors. Very sad. No time for innovation or upgrades anymore. I guess this is what happens to all small companies. And yes, of course, the parent company uses MSFT. I'm happy to say their R&D engineers saw how we do things with Notes and the web. Impressed. That made me feel great. Their IT managers opinions are the ones that count thought. No future Notes apps. They want to provide employees with AD, Outlook, Office, and MSIE. Why do users need anything else? Lock down the user desktops via AD policies. I guess this is why the parent company had to buy a little company like ours. Their R&D groups have no freedom or innovation.
In smaller companies, IT staff are there to provide "end user support". Help the users be innovative.
In larger companies? IT staff get to dictate policy. This is not "end user support".
Comment posted by Henry Ferlauto04/14/2008 02:31:49 PM
Homepage: http://www.geniusinside.com
I think one of the keys will be the ISV community. I would not recommend the "forceful" approach of having the next version of their software absolutely require ND8, because then that vendor is probably hurting themselves in the process. However, if the application had features that were enhanced if installed in an ND8 environment then the shop may be more enticed to move to ND8.
Or in other words, use a carrot, not a stick.
Comment posted by Chris Whisonant04/14/2008 04:56:53 PM
Homepage: http://www.bleedyellow.com/blogs/lotusnut
I mostly see R7 servers to be honest. Many customers have a mixed bag with the messaging servers usually being the first to be updated and maybe keeping app servers at 6.5.x. As some have also mentioned, there are a lot of 3rd-party apps that don't work with 8 yet. I recently had a customer who had already moved to 8 but had to later downgrade because of a product purchased after the upgrade that isn't supported on 8 yet.
I, too, have to ask why you're comparing the short Domino server release cycles with the big gaps in Exchange releases. Between Exchange 5.5 and 2003, there were, what, 3 major Domino releases in that timeframe? You're comparing apples to oranges here (considering that Exchange/Domino comparisons are similar in the first place).
For a Domino blogger, you sure have been talking a LOT about Exchange lately. Good grief... and you talk about us talking about Exchange too much...
Comment posted by Bruce04/14/2008 05:18:48 PM
@Chris,
So with all the updates coming out of Lotus many customers are skipping over certain releases. How do you explain that? I am not obsessed with Microsoft as you are . I am simply trying to say that Domino shops typically follow the same upgrade patterns as MS shops do. Many MS shops skipped over Exchange 2000 went to 2003. Many Domino shops on 6.5 are going to skip 7 and go to 8.
Comment posted by Bruce04/14/2008 05:22:57 PM
@Chris,
And one more thing....
I know how to setup, configure and administer Exchange. I also don't ever bash Microsoft or Exchange. There's a difference about talking about Microsoft and "knowing" about Microsoft Exchange.
I speak from experience not marketing FUD.
Comment posted by David Killingsworth04/15/2008 01:12:02 AM
Homepage: http://domino.symetrikdesign.com
1) Because it's not broken. 6.5.x seems pretty stable, or at least our user base is used to it.
2) There is no training function at our company of 2500 employees. Training on how to use 6.5.x is word of mouth (and we are using the OpenNTF1.7b template). Getting everyone to the next version is a huge undertaking just in the user acceptance level.
3) Domino was implemented by administrators who are no longer with the company and the education to implement Domino policies and Smart Upgrade was never thought of until experienced admins came into the company. By that point, we are afraid to implement Smart Upgrade because half of the company has single install Lotus Notes, and half the company has shared install. We are not sure of what Smart Upgrade will do.
4) We have heard that Smart Upgrade doesn't work as well as it's advertised.
5) We are piloting R8.01 (server and client) as administrators. There are 5 of us, and we have been working closely with IBM. We have an issues register which lists 30 issues/problems that we have found as administrators with the client.
6) IBM has told us that no other company of our size in Hong Kong has migrated to R8 yet.
7) We can't justify upgrading 2500 desktops with 1GB of memory.
8) We just don't see that the R8 Standard client is ready for installation on most of our desktop hardware. It's sluggish even with 1GB of memory. Much of our desktop hardware needs to be updated, but it's a cost that the business is not willing to fund just to get a new version of the software working.
9) End User acceptance of the R6.5.x GUI by some business units is so bad that they refuse to use it (that have outlook to POP3 on Domino, outlook connector, and DAMO installed). When showing them R8, we are asked about the costs of upgrading hardware to get R8 working smoothly versus the cost to just move to Exchange.
10) See number 8. If we are going to go through such a large change on the client side (which is all the business sees) and the user training, and the cost, why not just go to Exchange...which is what they have wanted all along. They don't realize or care about what takes place on the back-end.
Comment posted by The Unofficial Poster Child For Lotus Notes and DOmino04/15/2008 03:29:56 AM
Homepage: http://www.bobbaehr.com
Greetings:
The decision to upgrade can be easily explained in mathematical terms:
Money (to pay for the upgrade)
Upgrades cost money, and the amount of money varies based on many factors (number and location of servers, operating systems, etc.)
Time (to do the upgrade)
Time is also a variable, because it takes longer to upgrade 10 servers than it does 2.
Work (involved in the upgrade)
Work is also a variable, because it takes more work to do the upgrade of 10 servers than it does 2.
Power (to approve and schedule the upgrade)
Power is a simple variable - yes, or no!
Knowledge (of the upgrade process)
Knowledge of the upggrade is 'fairly' constant - you must know Domino Administration and the upgrade process to be successful.
So, where am I going with this?
We all know that "Time is Money"
We also know that ""Knowledge is Power"
In mathematical terms, this results in
Time = Money
Knowledge = Power
So, using the standard equation for Power (to do the upgrade), we get
Power = Work / Time
Now, substituting Money for Time, and Knowledge for Power, we get
Knowledge = Work / Money
Therefore, using basic algebra, solving for Work (the upgrade), we get
Work = (Knowledge) * (Money)
So, we determined that Knowledge is 'fairly' constant, but, as Money approaches zero, the ability to do work (the upgrade) approaches zero.
Therefore, Money is the factor preventing Domino upgrades to 8.
As a "Proof", look at the line
Knowledge = Work / Money
So, the more money you make (Management), the less you have to know about the benefits of upgrading your Domino infrastructure!
Cheers
The Unofficial Poster Child For Lotus Notes and Domino
Comment posted by Chris Whisonant04/15/2008 05:39:57 AM
Homepage: http://www.bleedyellow.com/blogs/lotusnut
If we're talking client, there's no major reason to go from 6.5.x to 7.0.x unless there is a specific bug affecting the company (which I have seen). Other than that, the end users won't see too much change. So why not wait for that shiny new Hannover thing we had been hearing about for so long? (By the way, wish we had known it was such a RAM hog so that more companies could have been more prepared with 2GB machines...)
As far as the server, most companies probably haven't seen compelling reasons to upgrade. As many have said, 6.5.x works well and most admins who barely have time to admin may not be too excited about some of the additions with ND7 (like DDM and Fault Analysis - why oh why do they not upgrade to be able to do these things?!?) But I digress.
Back to the matter at hand - why do Domino shops wait a long time to upgrade like Exchange shops do?
This is the question you asked and I'll give you the answer...
Because Domino is much more meaningful to companies than Exchange is. (It's why Exchange people name their pets after Lotus products... )
Don't get all defensive, but learn from this truth Bruce: Domino isn't just a mail platform - companies run their entire business on top of Domino. It is their workflow system, their CRM system, POS system, Intranet, etc... oh yeah, and it runs their E-mail as well. Domino defines their company. (Hopefully there will be many companies that will run IdeaJam too for many compelling reasons! And I know that you know the value of Domino - just reiterating it...) So there are many who probably do have their hands tied with respect to upgrading as well as waiting on other vendors to certify products. And if that's the case, then they can wait for a couple of years before jumping versions.
Oh yeah, and they can wait generally resting assured that they won't have to rip and replace what they have. In some cases they'll have to upgrade an OS, but this newer OS will be supported by the version of Domino they're already on, so there won't be problems with that. Ahem...
Comment posted by Rick04/15/2008 10:50:17 AM
@ Chris - FWIW, I'm not sure that Bruce is the one being defensive here.
While I'm commenting, I'd also like th say that I'm not terribly fond of the argument that alternate platforms require a 'rip and replace'. Notes/Domino has much better value propositions that I think we should all focus on. Let's keep in mind that some folks running Solaris right now will only be angrier regarding this particular argument if they planned on moving to Quickr 8.1. If you don't like the Solaris example, there are others around Domino 8 and the i5 (aka AS400 or iSeries) that I could just as easily reference. To reiterate, I'm not a fan of the 'rip and replace' argument.
Also, you're kidding when you say "Domino defines their company", right?
Comment posted by Chris Whisonant04/15/2008 11:29:14 AM
Homepage: http://www.bleedyellow.com/blogs/lotusnut
@Rick - I'm not defensive here - I just know that Bruce won't like that argument though he hasn't rebutted it for me...
I understand that the "rip and replace" isn't the best argument, but as a Domino administrator it's one that I would find very compelling when looking into upgrades. I think the Solaris/Quickr thing is pretty lame on IBM's part, and Quickr for Domino doesn't even run on Linux yet. And it's somewhat off-topic since we're talking about Domino and Exchange and not what platforms either do or don't run on - especially what the extended products run. When was the last time Microsoft pulled support for Exchange running on Solaris?
And I assume you're talking about Domino 8 not running on V5R3? I pointed this out at my blog a long time ago. I don't like that either, but V5R4 has been out for a good while and I know personally what IBM's stance on supported i5/OS releases is. But those are looking over the fact that with all of the platforms IBM supports you don't have to upgrade hardware and OS just to upgrade your mail platform as you do with Microsoft. Again, it's the simplest argument to use because it's irrefutable.
And I'm fairly serious about how important Domino is to many businesses. When software is ingrained into the fabric of an organization, it's very difficult to just rip it out sometimes. Maybe my words didn't convey 110% of what I was trying to say, but I stand by what I meant.
Comment posted by Chris04/15/2008 11:38:39 AM
I know of a job opening in Atlanta that is paying $350k for an Exchange admin. Interested?
Comment posted by Charles Robinson04/15/2008 01:28:32 PM
Homepage: http://cubert-codepoet.blogspot.com
The reason I haven't upgraded to Domino 8 yet is because it requires RHEL5. Getting RHEL4 installed on an IBM BladeCenter HS20 connecting to an IBM DS4300 SAN was hellishly difficult due to driver issues. After that experience the network admin basically gave me the finger when I suggested we upgrade to RHEL5. 
As for Notes 8... I consider 8.0.1 Standard to be 8.0 beta 5. 'Nuff said. I'll probably deploy Notes 8.5 Basic about 3 - 4 months after 8.5 is released.
Comment posted by Tim Paque04/15/2008 02:05:08 PM
We use Rightfax, and Call Xpress and BES, and a handful of our users use NotesSQL.
BES is only updating now, (wow 8 months?) slowwww. NotesSQL, Not until 4th Quarter. And CallXpress and Rightfax?, If they offer R8 support, they sure don't make it obvious on their sites.
