holtaf 5 minutes ago

2/3

holtaf 6 minutes ago

23

holtaf 9 minutes ago

live 3p created, join up

SerOberynMartell 2 h 21 min ago

omriex are you here?

SerOberynMartell 2 h 24 min ago

Les sept couronnes waiting on you to ready up in live lobby

Soda-can 2 h 32 min ago

One more! I'm hostage baby.

Soda-can 2 h 39 min ago

3/6 LIVE

Soda-can 3 hours ago

It's really incredible to get to do this ²/6

Just do it 5 hours ago

3p live fast+

Daredevil Z 8 hours ago

3p live

CKyou2 10 hours ago

Ready?

CKyou2 10 hours ago

2/3

CKyou2 10 hours ago

1/3

Ihor 13 hours ago

2/3

Nicky 14 hours ago

Anyone 3p live?

Daredevil Z 16 hours ago

2/3

D_jaja 26 hours ago

5/6

omriex7 26 hours ago

1 more for 6p live everyone!

omriex7 26 hours ago

6p live 4/6

omriex7 26 hours ago

6p live 3\6

Did you support us already?



Community Forum
Search |  


Author
Message
JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 15:29
Dear fellas,

I just got off the phone with our server hosting provider - 30min call...
We totally ROCK the server!
which acually means we kill it and cut off other customers sharing the same hardware.

Users' activities sky rocked during the last 2 months. Presumably due to the World Cup that is running at them moment.
In addition, I know at least from 2-3 coders, there are external script activities to crawl game information for statistics.

To sum it up, we need more resources or change the way to access games and data. User activity limitations, geo blocking, IP blocking, anti DOS measures, pay per use...
If we cannot manage to maintain the status quo regarding server requirements, the smallest package we are allowed to use is 10 times more expensive comparing to what we are paying now!

That is why our server is moved temporalily to an isolated server.
This happen TONIGHT at 00:00 UTC or 2am CET (european time). Expected downtime: 1-2 hours!


IMPORTANT:
We have 1 1/2 weeks to figure out solutions. I am on vacation for 2 weeks after that. After that we pay 10th, have a solution or no server at all.

best to you all,
JM Grip


Leonhart
Blacksmith

Posts: 5
Games: 141
Rank Points: 139
Member since: 2016-Feb-05

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 15:49
Hi JM,

Shame to hear that but not unexpected in my opinion.

Are you able to give insight on the server requirements in terms of what is required? Have you looked at other hosts?

Worst case scenario I think a monthly / annual charge may be the answer..

Mike
SKJakubekk
Knight

Posts: 4
Games: 146
Rank Points: 740
Member since: 2018-Sep-26

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 16:28
I guess we could donate some dollars.. there are many people if 50% donates some amount it would be great i think :/


Lannie85
Protector Of The Realm

Posts: 6,063
Games: 871
Rank Points: 6,222
Member since: 2014-Feb-25

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 16:44
Fundraiser!


Ser Hodor
Son Of Hodor

Posts: 756
Games: 1,331
Rank Points: 10,054
Member since: 2016-Mar-20

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 17:23
I sent some through Paypal this morning.  Valar Dohaeris.
ElPopelos
Keeper of the Forbidden Wilds

Posts: 3,874
Games: 179
Rank Points: 1,271
Member since: 2013-Feb-01

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 20:37
While having increased DOnations, i dont think that this is the way to go on the long run. 10 times the current amount is just way too much.

While the statistics show that this peak wont last forever, we have to plan with a bigger playerbase overall.

Limiting the possibilities for players or excluding certain regions also cant be the way to go, or this Community would die slowly due to a lack of development.


Im no expert in this (not even a novice), but i would be curious about a few options:

1) We had a similar peak 2 years ago. What changed that we have problems now?

2) Blocking any external scripts and only allow single use of them with direct permissions (like people have to ask before they can fetch the data). How much overall does this make though? Is this worth the extra effort?

3) Are there other servers avaiable which would be cheaper but have the same value?

4) I might be wrong here, but is it possible that a lot of traffic is generated due to automatic statistics (like the ones on player profiles and so on)? In that case, is is possible to change the way that is done? For example generate the data on request and save them up and whenever a new request comes up, only the new data between the old and the new request need to be fetched again. I could imagine that this saves the sever a lot of work?


JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-20 20:46
I am already looking for alternatives. We could her dedicated servers for only 3 to 5 times the amount we need now. But moving to another provider is always more work.
Which I cannot do in a few days.


JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-21 21:50
logged main access within the last 12 hours. We had 700k hits, website not counting.
That makes at least 16-17 hits per second, with website hits presumably 20-23 hits per seconds on average. That is a lot.
primetime in the evening can reach up tp 50-80 hits per second...


Nomaris
Maester without a chain

Posts: 2,463
Games: 389
Rank Points: 2,591
Member since: 2017-May-19

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 01:58
Is it already clear in which part of the script the high traffic occurs (e.g. playing, (automated) looking at old games, automated statistics,...). If so, would it be possible to block that part to give us some time to come up with a good solution (assuming that it is not the actual playing, which is the problem)?


ajant
Blacksmith's Apprentice

Posts: 148
Games: 34
Rank Points: 88
Member since: 2014-May-31

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 09:55
i think ElPopelos is on the right track. if required statistics are provided officially, there would be no need to hit the site with scripts. i've never worked on data mining myself, but i assume there are users who have and would be willing to help

another solution might be to disable access to finished games data during the peak activity

i was thinking of writing some scripts myself, but never got around to it. i did ask JM what the best time to run the scripts would be however. if rest of the miners would do that, maybe that would be enough, but that's not very likely to happen i guess


JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 10:10
Can start analyzing today. My guess is that many people play more games at the end time or have multiple browser tabs lingering open. Each is requesting data every few seconds...

Normally I would say, the server must be able to handle it. But we try to work with minimal requirements and there is always something you can code or a bad implements script.


JaqenHghar
Faceless Man

Posts: 108
Games: 453
Rank Points: 2,515
Member since: 2013-Aug-25

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 15:21
Since you brought up having multiple browser tabs open JM, I can tell you that whenever I play a live game I always keep the profile of each player I'm playing with and the chat commands open. That means, in a 6 player game, I keep at least 6 other tabs open simultaneously. Not sure if everyone does this, I do because if I would go offline, I would like to be able to reopen the game easily or try to message another player as easily/quickly as possible. If this behaviour is too taxing on the servers and prevalent maybe there could be a simple solution? Doesn't apply for PBEM games though and I question that live games are more popular than PBEM games.


JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 15:54
It is just the sum of all players. static html is no issue, but the shoutbox reloads in every window you have. same with each game, tab and chat.
I could disable auto-loading for now and implement a button for enabling it or something like that.

Thanks for the hint.


18HopeCSF
Sword of the Forest

Posts: 244
Games: 586
Rank Points: 3,309
Member since: 2019-Feb-09

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 20:34
I'm a complete ignorant of these kind of things so I don't know if this is relevant or not... but I've always noted that I need to log in several times a day on the same devices, and the same thing happens around quite a lot from what I've heard. I'm pretty sure that an important part of my activity on the site is just that, logging in. Could that be an issue?


JM Grip
The 8th God

Posts: 5,236
Games: 92
Rank Points: 686
Member since: 2007-Sep-28

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-22 23:34
I located some issues. Mostly game logs are slow... Hopefully can fix it tomorrow. Still working on some other minor things to reduce unnecessary requests.
Next week the provider's technicians have to evaluate if we can move back or not...


Nomaris
Maester without a chain

Posts: 2,463
Games: 389
Rank Points: 2,591
Member since: 2017-May-19

Topic: DOWNTIME TONIGHT!!!
Posted: 2019-Jun-23 00:03
I am a pbem only player and play mostly from mobile and thronemaster is almost always open in my browser (sometimes multiple tabs, since many links open new tabs). Could that also be an issue, even if I do not open the browser for hours, but only have it in the background?


This Topic is closed.

Please log in to use the reply function.
toast