Ed Brill

Two things come to mind.. firs tit sucks when anyone picks up an account and/or attempts to impersonate. While some of the smaller companies are fexible and will assist, larger ones like Google are growing too big and stay out of the legal issues with removing an account and proving identity. Who is to say two Bob Jones couldn't work at IBM and try to claim that address on Google? Who owns the rights?

But the other issue is Lotus' move into SAAS with Bluehouse. How will they be scaling? What technologies and SLA's exist for me moving users to that service? What if someone opens an account on there in my or my company name? Does IBM have a policy in place either way? Will IBM delete an account or company profile for me?

I am not just calling it out as a Bluehouse issue alone. Like I said ,all companies face this. It is more how an enterprise will handle this type of dispute with taking a side in deleting an account or proving someone is really who they portray.

My personal Gmail acct has been down for extended periods twice.

Once,suddenly after three years suddenly I couldn't log in and found out they had given my username to another user.

Recently my account got locked out. It took over a week to get it restored. There is noone to contact at all. I hope for their corporate customers thay actually have a help line.

@1 - Ironically, we had that very issue with your own identity on BleedYellow, didn't we? There was a Chris Miller already signed up. Except... that was probably a perfectly valid Chris Miller. So we had what I think was a nice solution -- but only because you had the power to reach out to me directly. I don't know that it would really scale.

IdeaJam has a decent solution for singlar names within a company space (I suggested it to Bruce.) When you have naming collisions within a customer, it's a challenge. The technology lets you deal with it pretty readily if you architect right -- but the process may not be scalable.

It's a serious challenge with SaaS. Google is neither the first nor the last to face it.

I'm curious how Dana Gardner knows that people have higher expectations of free, web-based email than they do their corporate email. I know that nobody within hearing distance at my company (many of whom use gmail personally, of course) complained when Google had their problems, but when Notes hiccups (and it does, occasionally) people go ballistic. So I question that particular statement.

My own personal opinion of the failure was that it was no big deal, partly because the service is free to me, and partly because it's still beta. If I paid good money, I might have a different opinion. Strangely (or maybe not), if Google's search engine went offline, I'd be horrified, stunned, and probably more than a little bit irate. Gmail down for an hour or two? Meh. And yes, it's my primary personal account.

In any event, I've never thought of gmail as suitable for enterprises. SMBs? Sure, that's a discussion worth having. And plenty of SMBs have more than 1 hour of downtime in a multi-year period. Heck, to be perfectly fair, gmail's reliability rating after that downtime still exceeds our internal Notes/Domino email reliability over the past year (no, I won't debate our implementation, it's owned by a different business unit).

The downtime, in and of itself, doesn't mean gmail isn't viable for businesses. The presence or absence of support, though? The business controls issues surrounding the 'cloud' approach? The security issues? Yep. There are definitely issues with both gmail specifically and 'cloud' or SAAS approaches in general. I just don't consider an hour of downtime to be one of them. Now, if that downtime occurs with regularity, and we keep seeing a whale picture when we go to gmail, that's a whole different ballgame (c'mon, we're talking about downtime and free public services, somebody has to mention Twitter)...

First up, "Enterprise" != "Business" is modern parlance. I run a business and I use Google Apps for my domain, with email. Works really well, and the price point is good. "Enterprise" has plenty of options in the whole email / groupware / whatever space -- small 1 - 10 man band businesses do not.

I think the relevant question here - and I can't see it referenced anywhere in the article - is whether the outage affected any companies using that "enterprise version of the service, at US$50 / user / year" or if it was only freebie users affected.

If it was only freebie users, well... tough luck; you don't get what you don't pay for.

But if it was those "Enterprise" paying customers too, then that really is food for thought.

Cheers,

- Mike

1) Since companies using a Saas model would not use @gmail.com email addresses -- yes, you can use Google Mail with your own domain name. Issues of namespace clashes / impersonation etc are *not* an argument against using Google as an enterprise level solution.

2) Is it confirmed that users of Google Apps were also affected? As far as I can tell, not all users were affected in the same way.

There are valid arguments against using Google or other outsourced services (eg privacy concerns, storage of sensitive data on servers within unpredictable justice systems, dependency on Internet uplinks for the probably most important internally used IT system etc), but the two examples Ed gives in his posting are not the real ones. Smells like attempted FUD.

I use Google Apps, and I didn't have an outage. I also have a separate normal gmail address, and that didn't receive mail for approx 20 mins. on Tuesday or whenever it was.

Are we saying that enterprise offerings never go down? Because I've experienced extended mail outages at various jobs / clients over the years (and yes, the worst offenders were on something other than Domino ;o) )

I guess you could sign this mail address up to every spam and pron service on the planet.... That would keep the mailbox full enough that it would dissuade someone from trying to impersonate you... After all, as we all know, no spam and junk filter is 100% accurate. It's just a matter of volume as to when it becomes intolerable.

As for the clustering thing Ed, I've said this every time I see you make this claim. The reason you see a comprehensive service is because you work in offline replica mode. In Online mode You DO see outages on clustered servers, many companies work this way for very good reasons. I have said this many times in many forums. There is no need for it but the development team, for some reason, see no need to recreate any "in flight" objects on the cluster server to provide a seamless service.

So whilst I agree that Lotus Notes/Domino in a clustered environment is significanlty better than almost all of the competition, it is by no means perfect and just because IBM works in offline replica mode does not mean that they do not need to supply a different solution to their customers. After all you have to dig pretty deep before the literature admits that you only get a seamless failover experience in offline mode.

Also before anyone tries to tell me I'm not training my users correclty, I suggest they reanalyse that statement from the perspective of continuous computing. You don't need to "Train" users to use real clustering and continuous computing. If you have to train them, then it is only part way there. Otherwise they could just have a second instance somewhere else.

As for the whole cluster.ncf scenario.. that's for another more technical forum.

For anyone who thought that Google was providing a fault tolerant service, I would recommend they refer to the age old adage "You get what you pay for". In this case you pay for products and those vendors advertise. Google takes some of this advertising money and produces "Free" services with it. If you do the financial breakdown, you are getting exactly what you pay for.

I have not heard that the subscription services went down. But even if they did, so what? I pay more for my MS Technet subscription each year and that is down more often that Gmail. I pay way more for my Yahoo services and they are offline more often than Gmail.

Was it just that we expected much more from Google for nothing? If so they have succeeded in a really big way.

@7 I didn't write the eWeek article, so it's pretty hard to accuse me of FUD. Also, while you *can* get a domain name on the Google enterprise offering, it's not a given that VSBs *will*. As Chris said @1, identity/authentication is a significant overall issue in the SaaS space.

@9 You seem to be saying that because Domino does offer a mode where any failover is seamless, but not everyone chooses that mode, that somehow Domino's flexibility == Domino is broken. All I'm saying is that with zero additional software investment, a 100% available Domino environment is possible, even if the way in which it is was not common architecturally. Is there any reason why it shouldn't be a common approach (while admittedly there are reasons for others) today?

I agree that you get you pay for here. It reminds me of Amtrak. If you catch the slow train to New York, they can delay you up to 12 hours without paying any compensation, where as the express, has to refund you the difference if it takes longer than the slow train. As a result, there is more motivation to keep the fast train on time and it consequently has much better performance.

For clustering, you also get what you pay for. If you pay to get it setup right, on good hardware, with good redundancy, and a solid design - it can be perfect(-ish). However, this costs way more than $50/year and it would involve some training int he mix.

We get lost in the Notes world a little too often. There are plenty of exchange shops that think nothing of an hour outage here or there (think Melissa, I love you, etc). For them Google's one hour is nothing - just means they have to pick up the phone instead.

Create a throw away address and ask the impersonator where to mail the 500 bucks you owe him from last year.

Any bets as to whether they take the bait? Does gmail still include the workstation IP address in the headers?

Sometimes it's nice having a somewhat generic last name.

@11 I do want to caution that Google's $50/user/year is not "all-in". IT still needs some staff, overhead, and connectivity on top of that.

@10, For failover to be truly seamless you really need to take the online case as the gold standard. However I though 8 solved that issue.

Another point on clustering is that it isn't available on all licensing options. So although Domino can do it, SMBs are attracted to a licence that does not allow clustering. You need to be careful you are not perceived to be pulling a bait and switch.

yes, express license != cluster license, it would be very nice feature for vsb's though, i know we would use it.

This is a very timely article for my particular situation and is already being used as ammunition in the "let's move to Google..." debates that are occurring. In our case, however, the debate is over MS Office licensing. Of course there is Symphony, OpenOffice, etc., but we also have alot of Citrix users and the additional investment into that farm to support the additional overhead is a big negative.

It's unfortunate, however, that there wasn't more put into clustering and failover technologies of Lotus, or even MS for that matter. "You get what you pay for" should be driven home.

@10 Ed, I did say Notes/Domino clustering was one of the best and I meant it.

However there is the fact, which cannot be denied, that IBM have made almost no effort to improve the client experience since I worked on my first HA Cluster 11 years ago back in 1997.

Yes you are exactly right, Notes/Domino CAN be seamless in failover. My take is it "Should Always" be seamless in failover. There is no reason why it should not be except, apparently, a lack of will to do so.

I was having a discussion today with guys about the Documentum failover process. The system rebuilds all the serialised objects on the failover node. It is quite standard, but IBM chooses not to do it. I still fail to have a reason why.

Then again Documentum replication sucks in a big way.... :-)

Ed, is this where I mention that with our clustered Domino environment we had only 15 mins of scheduled and unscheduled downtime from Nov 2001 - June 2007 and that one outage was only for 800 of our 8,000+ users? That continuous up time included 3 major version upgrades, a consolidation and relocation of all servers outside of the Metro DC area to our MD data center for all our regional offices (from 24 clustered regional servers down to 2) and two relocations of all Domino servers between data centers over 30 miles away from each other. 15 mins of downtime in 5+ years. Yes, you DO get what you pay for!

In Google Apps your company owns a domain. There is no EdBrillIBM@yourcompany.com problem. You own the domain, you resolve the issues.

Google Apps is not in beta. GMail is.

Google Apps did not have an outage. GMail did.

Can Domino have an outage. You bet it can. I have seen sites go down each night when a calendar task kicked in. All servers. At the same time. Can this be resolved? Yes. As can GMail outages.

The bigger issue is: will "enterprises" migrate to Google Apps? Yes, several are in the process. Their pain or not-pain will be important for the proliferation of this trend. It's too early to call.

All I can say is, migration is a good business to be in. There is a LOT of work. And you have to be in the camp that people migrate TO. If you are a Notes guy, you have to find work to migrate people TO Notes. You won't be trusted with the job to migrate people FROM Notes. Replace "Notes" with your favorite platform.

@19 so you are saying that 1) nobody who buys Google Apps uses the gmail.com domain 2) gMail being in beta after four years excuses the situation 3) Domino outages that are the result of architecture or implementation should be considered the same as system-wide outages of a service and 4) "several are in the process" of migrating to Google Apps, yet I don't know a single large business customer that is. OK.

No. I am saying what I am saying and not what you are saying.

1. If you buy Google Apps, you should use your own domain.

2. Free beer. Can't complain, or can you?

3. The Domino outage was system-wide. Every night. It does not matter to the customer that other customers are fine. The Gmail outage was once ytd. No free beer for a couple of hours. Paid beer was still on tap.

4. Your definition of large and my definition of large differ. My definition of large, just to make this clear is > 1000.

What's your definition of "several"?

Also the gMail outage wasn't once YTD. This is just the first one that got broad attention. Search twitter archives and other publications and you'll find other gmail outages YTD.

Yes but, yes but, yes but... GMAIL IS FREE.

Since when did free have SLAs?

@23 see { Link }

@10 - "Is there any reason why it shouldn't be a common approach (while admittedly there are reasons for others) today?"

For me it's two things. First, the asynchronous notification is buggy. There are technotes about it supposedly being fixed, but as of 8.0.1 it's still not accurate. Second, most of my Notes users are on Windows Terminal Services and I'm not going to invest in multiple terabytes of RAID storage just so I can get Domino clustering to work while still dealing with flaky asynchronous notifications.

@11 - Exchange users are not accustomed to extended downtime. They might have been about 10 years ago -- the era when I Love You and Melissa came out -- just as 10 years ago people expected Notes to be ugly and obtuse. Times change.

@17 - In Notes 8 if a server fails and a user is connected in online mode they are prompted to switch to another replica. I would call that a pretty significant improvement in the client experience. It should be completely seamless, though. Notes knows the primary server is down and Domino knows where the other server is. Why even ask the user what to do?

@20 - To answer your first question, you have to use your own domain with Google Apps. If you try signing up that becomes immediately apparent.

As for your last point, why is it that any time anyone mentions migrations it's always in the context of large enterprises? Of the 25 million companies in the US approximately 10,000 have more than 1,000 employees. That's 0.04%. Do the remaining 99.96% not matter to IBM/Lotus? That is a rhetorical question, but I know the answer anyway.

@25 I tried to install the Notes8 standard client in the beta phase exactly twice. Both times were on a brand new vanilla XP Machine. Both times it failed and I lost interest which has never been revived.

Are you trying to tell me that in Notes8 when creating a document and a clustered server fails, then you will be prompted to change server on resubmit and it will re-create that document in exactly the correct context on the cluster partner server? That would be progress indeed because the R5 client prompted you to change server on a failure of a Clustered server when you tried to access the database.

My clustering experience spans 4.5 working with people who were part of NPN and also HACMP/Domino Clustering Mutual failover situations to D7 clusters. I grew up with Notes/Domino clustering and have worked on delivering it in many scenarios. Therefore I feel slightly qualified to comment on the issue.

The reason I challenge Ed every time I see him saying this is because of customer perception. When a customer buys the story of 24x7 continuous computing and then finds it's not true for "their" implementation or to their users expectations, they then disbelieve everything else they are told. Even if it is true. It is much better to explain the entire story up front and central before starting.

I have seen one site which reported 1,700 outages (helpdesk tickets), in a week for a cluster of Domino servers on AS/400. Go figure.

Charles, "large enterprise" is _the_ IBM customer. As long as I have been covering IBM, and that is more than 15 years I have been briefed _every_ _single_ _year_ on how they are now addressing SMB. Go figure. :-)

In my experience, large enterprises have high inertia on their current path. They will stay the course as long as it is possible. When they start moving in a different direction it comes at great cost and is therefore significant.

One example: Daimler-Benz was in the process of migrating to Exchange when they bought Chrysler. Chrysler was a Notes Shop. And they decided to go back to Notes midstream. The project almost failed but they put some serious management in charge which turned it around. Now that Daimler no longer owns Chrysler, you can place new bets.

What I found interesting when we put together the wiki with the world's largest companies was that there were no Notes wins outside of mergers like the aforementioned DaimlerChrysler scenario. I asked about that and was told that the wins that did happen were just not in exactly this population.

With Google Apps the current thinking seems to be that it ONLY happens with smaller more agile businesses. I generally agree with this idea. But I do also happen to know about large (as in > 1000 employees) companies in the process of migrating. How do I know? Through people who are working on the projects.

eWeek suggests that Google Apps is not enterprise ready. I think ultimately customers will decide. Google has found their "pragmatist in pain" and how well that works will define whether Google can cross the chasm.

@26 - I haven't tested creating a document, dropping the primary cluster server, then saving the document. I do know that if you're reading e-mail and the server drops it gives you a pop up and you just click OK to connect to a different server. It is a tremendous leap over what clustering was like in previous releases.

@27 - I understand from a business perspective why larger customers get more attention from Lotus. In a single move they can shift tens of millions of dollars to another vendor. What gets really annoying is the apparent belief by vendors and analysts that large companies are the bellwether small companies use to set their direction. It shows just how arrogant and out of touch both the big vendors and the analysts are.

Of course small companies want a vendor with a track record and staying power, and of course they want good support and a clear roadmap of a product's future. Do they really care that Exxon or Coca-Cola or DaimlerChrysler is using it? Not really. In fact, the small companies I know are immediately skeptical that anything used by a large company will have the features they need at a price they can afford. They want to talk to other companies *their own size* who implemented it. Or, better yet, have a very low barrier to entry for testing it themselves.

That's what makes GAFYD (Google Apps For Your Domain) so compelling for SMB's. There is no software to download or servers to configure and you can set it up in under an hour to see whether it is a good fit. The transitions you're seeing in the larger companies just tells me that they can be agile, too.

@25 Charles, you know that I tune in to both the enterprise and the SMB markets... I've spent days chasing a 50 seat "defend" situation in Florida this week, for example. There are 46,000 Notes customers, and more than 0.4% of them are <1000 employees. :)

Most of my notes customers have gone to Google Apps or the standards-based groupware Zimbra, either hosted with { Link } or in their own offices/data centers. I had one groupwise customer left, also went Zimbra. Even my Exchange customers are considering Google Apps or Zimbra now... I agree with the premise.

@26 - "I have seen one site which reported 1,700 outages (helpdesk tickets), in a week for a cluster of Domino servers on AS/400. Go figure."

That is a failure every 5.9 minutes, assuming a 7 day week. What is the rest of the story here ?

@26/31: So maybe 1700 users called the help desk? That begs the question should a WAN/LAN issue between the user and the Domino server be considered an outage? Most of those things are beyond the control of a Domino Admin, but NOT beyond the scope of a quality Domino implementation. That type of cusotmer-preceived outage will be one thing to keep in mind when these internet based systems are unreachable for an entire enterprise becauase their corporate F/W or WAN has an issue... after which Google says, "Hey we were up!" Time to check that SLA again?

@31/32 yes the story is that users in the hundred+ called the helpesk every day as their servers were going down during mail creation and they had to re-create the mail. Something you don't have to do when working offline. They had been sold continuous computing when the reality in the situation they were working was different.

Also we had some fun issues with users who came back from holiday and their client didn't fail over when their mail server was down. No cluster.ncf, no resident info, no failover. Not very fair to log it that way but that's life with users.

It is explicitly for these reasons that I comment when Ed does the "Clustered Notes/Domino never goes away IBM is living proof".

This isn't about the technology per-se, it's about managing user perceptions and getting the best out of a good tool. If you always sell the best possible scenario, then you will be pilloried when the worst scenario comes up.

I think so. You cann't store everything on google's server, do you know how long it takes for google to give you a reply on something. Most of us are not that famous as google, and google will not take much care of you, if something wrong, just something wrong.

Keep your information yourself

1 year ago I post this question

{ Link }

Too few answers for a final verdict

Those entities who place faith and trust in Google to provide care, custody and control of their private Corporate email or application data should read this:

{ Link }

In short, Google is stating that there is no right to privacy when they drive up your driveway, which in the case in question is on a private road, and film the public areas of your home for the 360 degree street view. That may well be a legally supportable position as described in the above link, but certainly does not build the kind of comfort and trust that would make large corporations with legal matters feel good about them hosting their email or applications.

That simply isn't true.

We use SalesForce.com here and scarcely a weekend goes by without an announcement of routine maintenance and an expected time window for an outage. Not infrequently there have been unplanned outages too.

I can only recall one significant Google outage - that recent one - and it didn't last very long.

Google Apps isn't yet the answer for enterprises, but it isn't a lack of reliability which is the main barrier. It is compliance with Data Protection legislation and user acceptance that conspire to rule it out.

There is no hosted solution today, that is perfect. There is no on-premises solution that is perfect. It's a balancing act of many factors and pain points.

"The issue was caused by a temporary outage in our contacts system that was preventing Gmail from loading properly. Everything should be back to normal by the time you read this."

So, human error. Who has seen a software bug that ended up being QA'd on the customer?? Any one? Come on - raise your hand. Yes, you.

As far as up time is concerned, my largest client has over 200 Domino mail servers that are all clustered. We get about 99.98% up time (we are supposed to have 5 9's). Most outtages are due to third party software, namely McAfee, but it happens. If we remove anti-virus from the mail servers, our up time goes up.

Do I think Google is ready for the enterprise? Not yet.

I highly recommend Salesforce.com. Their developers that do custom work haven't been the best, but the app catalog they have is rock solid as well as their service and up time.