[Molly]

After waiting 2 weeks in vain for the crashed server to come up again, (and for the server people to even respond to our e-mails and phone calls), we’ve decided to cut our losses and start looking forward again.
The gameplay has fortunately been affected very little. We moved to a new and better server over a week ago, and we had fresh backups of both the code and the player files.
We are however going to lose a month’s building work on all three Ports, since the last backup of the World files is from November 5th.
Worse still, we’ve lost our Webpage and our Discussion Forums. We’ll have to restore the webpage from scratch, which means extensive work, and the Forums are lost for good, unless the server decides to come to life again.
We also seem to have lost about half our playerbase, but hopefully most of them will find their way back, once things return to normal. The old address, 4dimensions.org Port 6000 works again.
All this is a pain in the behind, but it takes more than that to bring us down. And no use crying over spilled milk.
On the bright side, the work on our newest feature, the Gladiator Arena, has proceeded during all this, and we’re going to open it for playtesting within a week. I’ll post some info about that, once we are all set to go.

[Muirdach]

I'm sorry to hear that, it's always so frustrating when the hardware lets you down and there's nothing you can do about it. As a creative type, my biggest fear is that something technical will fail either in the hardware or code, leaving everyone's effort useless until it can get fixed. Glad to hear that you're still going, and I hope you can get your players back. If there's a silver lining, maybe you'll end up with more reliable hardware and hopefully better service.

[Kylotan]

Please, spend some time writing a backup script or something. In this day and age, there's no bandwidth or CPU time obstacle to zipping up all your areas and sending them to someone on a nightly basis. I hope it all gets back to normal, anyway.

[Molly]

Originally Posted by Kylotan Please, spend some time writing a backup script or something. In this day and age, there's no bandwidth or CPU time obstacle to zipping up all your areas and sending them to someone on a nightly basis. I hope it all gets back to normal, anyway.

We actually do have a backup script that zips up and sends the code and pfiles daily, which is why the gameplay wasn't affected, except by the downtime. Doing the same for our very large world seemed a bit over-extended, especially when you pay for a server that is supposed to keep daily backups. Even zipped, the world files take up almost 20,000,000 and that is a bit much to handle on a daily basis, at least for my computer. We did take safety backups of the world files ourselves once a month, however, and that's what saved us from total disaster. We'll probably do it once a week from now on.

It would actually be interesting to know how many Muds that take daily backups of their world files themselves, (I mean, apart from the ones that the server is doing).

The server is still down, and we still don't know what happened, so I doubt we'll get any help from there.

[Kylotan]

I find it very hard to imagine why you need 20 megabytes for the world! English text compresses very, very efficiently and most world data is just that, plain text. (Obviously it's not easy for me to see how many rooms you have or if you have any other weird data that doesn't compress well.) If it really was a bit cumbersome then such a script could just opt to send the files that changed that day, which I expect is only a small subset. Something to consider for the future, anyway.

[Molly]

Originally Posted by Kylotan I find it very hard to imagine why you need 20 megabytes for the world!

Well, I'm just looking at the compressed files I have as backups for our three Ports in my own computer. The one for the Port 6000 says 10,417,902, and the ones for 6001 and 6002 are 10,267,203 and 4,905,652 resp.. (6001 is pretty much a duplicate of the Game Port, but a lot of building and script testing goes on there, since we don't like editing in the game Port. And 6002 is the official building Port, where most of the new zones are created).

We have 227 zones with 70100 rooms, 20983 mobiles (8202 prototypes), 32013 objects (12450 prototypes) and 6521 elaborate triggers in the Game Port. Plus 378 zones with 71003 rooms under development in the Build Port (most of which are incomplete, and some are abandoned, of course). .
I guess it adds up. We have after all been on line for almost 10 years.

I guess we could confine the backups to zones that have been touched in the last day, but since typos and stuff are corrected continuously from typo and bug reports, it would still affect a lot of the zones. Small changes, but still changes. What really sucks is that the active Builders get hurt the most, while the inactive ones won't even notice.

And the thing is, you just don't expect anything like this to happen when you are paying for a commercial server.

[the_logos]

Originally Posted by Molly And the thing is, you just don't expect anything like this to happen when you are paying for a commercial server.

Unfortunately, you should, as we ourselves learned a few years ago while hosting with a MUD-specific provider. The problem is that your downside when something goes wrong is SO much larger than the hosting company's, as they don't typically incur a lot of penalty for losing your data (and none for the loss of income associated with being down if you're commercial). They're less motivated than you are to prevent 100% of problems. Luckily, we were able to get our harddrives from the ISP and send them out to a data recovery firm (they had crashed, and the host had failed to actually run the backups that were their responsibility) and were only down for 6 days, but it was horrible.

Our feeling now is: Investigate your ISP thoroughly and then trust, but verify. And run external backups along side the internal backups. (We do a triple layer of backups now as I'll be damned if that's ever happening again!) You should have no problem doing at least weekly offsite backups for instance, which will minimize any future problems like this, if not completely get rid of them.

--matt

[Newworlds]

Originally Posted by Kylotan I find it very hard to imagine why you need 20 megabytes for the world! English text compresses very, very efficiently and most world data is just that, plain text.

Are you kidding me? Since when does text compress well. Text is likely the worst compression of anything because of the lack of duplicate data structures. At best you'll get around 50 % compression and when dealing with megabytes that isn't much good. (Compare this to 90% on graphic and sound files).

Aside from that, who would backup nightly player files. I have to laugh at this because it sounds so over the top unless you are a commercial enterprise promising such rigorous redundancy.

NW is an "offline" building system so all of our game files are on no less than 4 computers continually so any complete server shutdown would result in zero lost worked. I'm not sure what game you are using Kylotan, but NW consists of over 50 megabytes in game file data. All of it text files and to back this up nightly would be notoriously wasteful.

Anyways, back to Molly and her game. I'm sorry this happened to you Molly, I like 4-D, tell your players to relax, a few days without the game isn't going to kill them. I was on a game once that lost 2 months of player files. Who cares, it's a game!

[the_logos]

Originally Posted by Newworlds Who cares, it's a game!

That's true, but that sentiment also unduly minimizes the players' experience.

If you've ever been really into anything optional, whether it be a sport, a MUD, a hobby, an over-paying job, or whatever, you can always minimize it by just throwing off this nihilistic 'who cares' attitude, but the fact is: Your players care.

That's why they're playing, after all: Because they deeply, deeply give a sh*t, and it behooves admins to treat that feeling with a great deal of respect.

--matt

[Molly]

Originally Posted by Newworlds Aside from that, who would backup nightly player files. I have to laugh at this because it sounds so over the top unless you are a commercial enterprise promising such rigorous redundancy. <snip> tell your players to relax, a few days without the game isn't going to kill them. I was on a game once that lost 2 months of player files. Who cares, it's a game!

Heh, you must have a lot more patient players than we do, if you can say that

Then again, I'd say it depends on the nature of the game. If the focus is on roleplay, then out-of-date pfiles possibly wouldn't matter all that much. But 4D has a strong amount of hack'n'slash and is very competitive. A lot can happen in one day, if the player is active, so to our players even the loss of one days exp and equipment would matter a lot.

Luckily we do backup player files nightly, so all they lost was 3 days downtime, until we got on the new server. It may be 'just a game', but we take it seriously - and so do our players.

The_logos is right, players do care, and they do deserve some respect.

[xotl]

Best wishes for a speedy recovery and kudos on your tenacity Molly.

I use MudDrake Hosting :: Where the web comes to play (News) for Accursed Land's hosting and have been very happy with it, by the way.

It would be interesting to get a list of hosting sites up and have them rated.

[Kylotan]

Originally Posted by Newworlds Are you kidding me? Since when does text compress well. Text is likely the worst compression of anything because of the lack of duplicate data structures.

This is just plain wrong. Typical text is stored in 8 bits, allowing 256 characters of which we rarely use more than 60, half of which we only use once or twice per sentence, and the rest of which are used with very predictable frequencies. That is very amenable to compression. On top of this, most of it is predictable and repetitive. It turns out that the average English letter can be stored in about 2 bits on average, so that's about 75% compression rate already. Just because certain naive graphics and sound formats happen to compress better, doesn't make 75% bad, or anywhere near the worst. 75% compares favourably to mp3 compression rates, yet is lossless as well.

Since 4 Dimensions advertises itself as having at least 60,000 rooms, and 20MB is the compressed size, then I'd assume they're using about 1-1.5K of data per room, which is quite high for a Diku-derivative. Perhaps their builders put more effort than usual into each room/object/mobile; even more reason to back it up frequently.

Aside from that, who would backup nightly player files. I have to laugh at this because it sounds so over the top unless you are a commercial enterprise promising such rigorous redundancy.

I wouldn't. But it's obviously important to Molly, and presumably to their players. Who are you to mock this choice, really?

I was on a game once that lost 2 months of player files. Who cares, it's a game!

The players care. If they didn't, they wouldn't play. And one of the things they care about specifically in such games is the persistence of their character.

[Muirdach]

Originally Posted by Newworlds Anyways, back to Molly and her game. I'm sorry this happened to you Molly, I like 4-D, tell your players to relax, a few days without the game isn't going to kill them. I was on a game once that lost 2 months of player files. Who cares, it's a game!

It may not kill them, but it could seriously harm the game. A dedicated player could play upwards of 5 hours a day. A week of downtime means that the player is spending those 35 hours on something else - probably another game, or at least searching for other games. That in itself is already quite an investment that would allow you to get pretty far in most games - they could well find something that they like just as much, if not better. The longer it goes on, the less likely they are to come back, as at some point when they have found another happy place and don't even know for sure if their old game will ever come back (this happens fast, people are impatient), they'll stop checking.

Players expect loyalty to go both ways. They expect, rightly or wrongly, that if they are loyal to the game then that game will always be there for them to play. If there is regular downtime, they start feeling that they are not wanted enough for the owner to spend some time and effort to make sure that the game is available in a stable manner. Game or no game, dedicated players spend several thousand hours on each character. That's years of someone's leisure time, spent in the expectation that their character will continue to grow, or at the least that they will be able to continue as a player for an indefinite amount of time. Maybe it's an unrealistic expectation for the average small hobby game, but they care enough to spend that amount of time, and time is money.

One of the things that MUDs have a real advantage over graphical type games is the persistence - if you like the text genre, it never becomes dated or obsolete, as long as development continues. People play MUDs for the future, not the present. They expect to be playing and able to play, 2, 3, 4, 5 years from now. If they didn't believe that they would be, there would be a lot fewer players, guaranteed.

[Newworlds]

Originally Posted by the_logos That's true, but that sentiment also unduly minimizes the players' experience. If you've ever been really into anything optional, whether it be a sport, a MUD, a hobby, an over-paying job, or whatever, you can always minimize it by just throwing off this nihilistic 'who cares' attitude, but the fact is: Your players care. That's why they're playing, after all: Because they deeply, deeply give a sh*t, and it behooves admins to treat that feeling with a great deal of respect. --matt

No Matt. Your players care. That's why they are "paying." Let's not mix apples and oranges. Your entire comment here sounds like you are running for office.

[Newworlds]

Originally Posted by Kylotan This is just plain wrong. Typical text is stored in 8 bits, allowing 256 characters of which we rarely use more than 60, half of which we only use once or twice per sentence, and the rest of which are used with very predictable frequencies. That is very amenable to compression. On top of this, most of it is predictable and repetitive. It turns out that the average English letter can be stored in about 2 bits on average, so that's about 75% compression rate already. Just because certain naive graphics and sound formats happen to compress better, doesn't make 75% bad, or anywhere near the worst. 75% compares favourably to mp3 compression rates, yet is lossless as well.

Mayhaps you are using a different compressor. I use winzip and the best I get out of it is 45% compression on text.

[Newworlds]

Originally Posted by Molly Heh, you must have a lot more patient players than we do, if you can say that Then again, I'd say it depends on the nature of the game. If the focus is on roleplay, then out-of-date pfiles possibly wouldn't matter all that much. But 4D has a strong amount of hack'n'slash and is very competitive. A lot can happen in one day, if the player is active, so to our players even the loss of one days exp and equipment would matter a lot. Luckily we do backup player files nightly, so all they lost was 3 days downtime, until we got on the new server. It may be 'just a game', but we take it seriously - and so do our players.

NW is based on RP mostly, but I'm sure if we went down for two weeks all of the players would go nuts. The most we've ever been down in the last four years is 6 hours in the middle of the night once. But never lost player files or any files before.

[Samson]

Originally Posted by Newworlds Mayhaps you are using a different compressor. I use winzip and the best I get out of it is 45% compression on text.

I use winzip as well and with nothing but text I've consistently seen 90%+ compression rates on it.

So unless tar.gz files end up with radically different stats, they should do just as well and Molly should be able to produce a working backup file that can be downloaded weekly without causing too much disruption.

[Lasher]

MCCP Compression summary for players currently online:

Raw bytes: 2,097,075,723
Compressed: 287,473,181

Doesn't really represent any specific time period as it varies for each individual, but is a pretty typical sample.

[scandum]

Mud output is generally repetitive though, often 50% of it is the same prompt (over and over) so zlib compresses it well.


Tarring using bzip2, (tar -jcf on most machines) rather than gzip, might create smaller backups btw.

[Davairus]

This almost happened to our game too. We got served 24 hours notice the server was done with mud hosting. We thought we had backups done via rsync to a server in australia, but when we asked the guy for them, it turned out it stopped working months ago and he hadn't said a thing. The rest of our backups were on the server that was shutting down. What can you do really, download it to your own computer? Well, we had a last minute rush to download the entire mud and get it up someplace else. Wasn't even much time to shop around for a new place. Service at wolfpaw has been wonderful, but there's always the worry that he'll someday decide to pull the plug, after you've been through something like that. Oh well, good luck with the new place. Suggest you tell your builders to copy EVERYTHING they do into notepad. Thats what I do. When work is lost its kind of their fault, because I will have said that til I'm blue in the face - never take the mud's existence for granted when it comes to storing something of value. Its been a lot more effective for me than trying to maintain some state-of-the-art backup system (i.e. the rsync thing). Pfiles is the only thing we really need to backup regularly "just in case".

[KaVir]

Originally Posted by Davairus This almost happened to our game too. We got served 24 hours notice the server was done with mud hosting. We thought we had backups done via rsync to a server in australia, but when we asked the guy for them, it turned out it stopped working months ago and he hadn't said a thing. The rest of our backups were on the server that was shutting down. What can you do really, download it to your own computer?

I make personal backups of my code every day, but back in October 2004 I managed to lose a 1500+ line code module when the Gryphonmud hosting service vanished overnight. I was warned by a Gryphonmud admin that I had 24 hours to grab whatever I needed (as they'd paid until the end of the month), but Inetsolve (which Gryphonmud was using) just pulled the plug on the machine about an hour later.

The problem was I'd forgotten to use the '-d' (directory) option when I'd grabbed the most recent version of the code from CVS, so it hadn't backed up the new directory for the module I'd recently added. It cost me four days to rewrite it all, so it's not a mistake I'd make again.

[the_logos]

Originally Posted by Newworlds No Matt. Your players care. That's why they are "paying." Let's not mix apples and oranges.

Nonsense. There should be as little difference as is financially possible between how a hobbyist treats players and how a commercial enterprise treats players (note that whether the players are paying or not is a bit irrelevant. Facebook is free to the user but I guarantee you that Mark Zuckerberg - its CEO - cares a lot about the experience he delivers to his users).

Why should there be so little difference? Because in the end your players have to pay to play one way or another. They have limited time and they have chosen to spend their free time on your MUD rather than the bazillion other options they have for spending their free time. An admin who doesn't respect that is a bad admin.

--matt

[Newworlds]

I removed this post because after reading it later it seemed very trite and much too agressive against the_logos, with little impact.

If I wasn't supposed to do that let me know. I have a copy and can repost it.

[Davairus]

In between the little scuffle emerging there, I wanted to add that server owners should keep in mind that some of their mudding audience is people still running ancient pentium-2 pc's with 200 mb disk space that can't even run diablo 2 smoothly - on dial-up. From Europe!

[Milawe]

Sorry about the delayed reply. I've been out of the country.

I'm sorry to hear about this happening to any mud! It's a terrible, terrible thing to lose creative material since it's impossible to replace or duplicate. I've been devasted when I lost just a few lines of writing from a freak crash or a power outage. I can't imagine losing major parts of a MUD.

Good luck in recovery!

[Muirdach]

Originally Posted by Davairus In between the little scuffle emerging there, I wanted to add that server owners should keep in mind that some of their mudding audience is people still running ancient pentium-2 pc's with 200 mb disk space that can't even run diablo 2 smoothly - on dial-up. From Europe!

Actually, and I don't mean to spawn a whole 'nother thread here, but this scenario may become more and more similar as technology advances. That may make no sense, but as PDAs, city-wide free wireless networks and cheaper cellphones/plans become more advanced and common, more and more people are going to be gaming from such devices. I think, in the near future, many mudders will no longer be playing from the big gamer's PC with a high speed dedicated connection, but on-the-go from these smaller and less powerful pieces of machinery.

[Kylotan]

It may well be true, but I never did see what relevance it had to this thread.

[Newworlds]

Originally Posted by Muirdach Actually, and I don't mean to spawn a whole 'nother thread here, but this scenario may become more and more similar as technology advances. That may make no sense, but as PDAs, city-wide free wireless networks and cheaper cellphones/plans become more advanced and common, more and more people are going to be gaming from such devices. I think, in the near future, many mudders will no longer be playing from the big gamer's PC with a high speed dedicated connection, but on-the-go from these smaller and less powerful pieces of machinery.

Relevant or not, we have at least five players on NW that play the MUD from their cell phones or blackberries. Amazes the hell out of me. Can't wait to get the NW Staff blackberries so they can manage from anywhere. Go Technology!

Incidentally, when I asked them about how hard it is, the common complaint is that typing is a biatch, so we are working on enhancements for cell phone players to make it faster and easier for them similar to our blind player enhancements. Who says TEXT MUDs don't advance with the times.