Project

General

Profile

Actions

Defect/Bug #94

closed

Grief Prevention Loading Time at 5 min. - B-Team

Added by Slind over 9 years ago. Updated over 9 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
2015-06-09
Due date:
2015-06-12
% Done:

100%

Estimated time:
5.00 h
Resolution:
Fixed/Completed
Modpack(s):
Attack of the B-Team
Node:

Description

The claim loading time on B-Team is extremely high. We should either look for an automated cleanup of claims from players that didn't spent much time or high improve the performance of the loading process.

Comparison:

B-Tema 1 (flat file)
    loading time     6m 32s
    claim data    6.421
    player data    34.694

B-Team 2 (flat file)
    loading time     6m 4s
    claim data    7.571
    player data    28.947

B-Team 3 (mysql)
    loading time     35s
    claim data    2.388
    player data    7.374

Script for flat file to db migration (not needed but at least 10 times faster than grief prevention's, but it is ignoring parent ships and non builder trust's, which are barely used) - created as grief prevention didn't want to convert in the first place

#!/usr/bin/python

import MySQLdb
import os

db = MySQLdb.connect("localhost","user","password","database")
cursor = db.cursor()

for fn in os.listdir('./plugins/GriefPrevention/ClaimData'):
    claim_id = fn
    claim_data = open('./plugins/GriefPrevention/ClaimData/' + fn, 'r')
    lines = claim_data.readlines()
    claim_data.close()

    if len(lines) > 4:
        claim_lessercorner = lines[0]
        claim_greatercorner = lines[1]
        claim_owner = lines[2]
        claim_builders = lines[3]

    cursor.execute("""INSERT INTO griefprevention_claimdata (id, owner, lessercorner, greatercorner, builders, containers, accessors, managers, parentid, neverdelete)
                          VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""", (claim_id, claim_owner, claim_lessercorner, claim_greatercorner, claim_builders, "", "", "", -1, 0))
    db.commit()
    print (claim_owner)

for pn in os.listdir('./plugins/GriefPrevention/PlayerData'):
    player_name = pn
    player_data = open('./plugins/GriefPrevention/PlayerData/' + pn, 'r')
    lines = player_data.readlines()
    player_data.close()

    if len(lines) > 4:
        player_lastlogin = lines[0]
        player_accruedblocks = lines[1]
        player_bonusblocks = lines[2]

    cursor.execute("""INSERT INTO griefprevention_playerdata (name, lastlogin, accruedblocks, bonusblocks, clearonjoin)
                          VALUES(%s, %s, %s, %s, %s)""", (player_name, player_lastlogin, player_accruedblocks, player_bonusblocks, 0))
    db.commit()
    print (player_lastlogin)

db.close()

DB cleanup queries

DELETE
FROM `griefprevention_playerdata` 
WHERE (accruedblocks = 200 AND bonusblocks = 0) OR lastlogin < NOW() - INTERVAL 14 DAY;

DELETE c
FROM `griefprevention_claimdata` c
LEFT JOIN griefprevention_playerdata p
ON c.owner = p.name
WHERE c.builders = "" AND p.name IS NULL;

Actions #1

Updated by Xfel11 over 9 years ago

Which bteam server are we talking about here? bteam 1 and 2 use a flatfile datastore, bteam3 is using a mysql data store. Which of these is the problem?

Actions #2

Updated by Slind over 9 years ago

1 and 2 I'm sure of, 3 I need to profile.

Actions #3

Updated by Xfel11 over 9 years ago

So it's probably related to the poor data model of the flatfile data store. One flaw I could identify is the way it loads the claims for one world: it actually has to open all claim files once in order to find out if they are in the currently loaded world, and if the world matches, the file is opened again to actually load the claim. It might be possible to optimize this.

Actions #4

Updated by Slind over 9 years ago

Maybe we can port it from the 1.8 build as I never noticed any issues on there.

Actions #5

Updated by Slind over 9 years ago

BTW. It feels to be exponential, but I have no proof.

Actions #6

Updated by Slind over 9 years ago

  • Description updated (diff)

Comparison:

B-Tema 1 (flat file)
    loading time     6m 32s
    claim data        6.421
    player data        34.694

B-Team 2 (flat file)
    loading time     6m 4s
    claim data        7.571
    player data        28.947

B-Team 3 (mysql)
    loading time     35s
    claim data        2388
    player data        7374
Actions #7

Updated by Slind over 9 years ago

  • Description updated (diff)
Actions #8

Updated by Slind over 9 years ago

  • Description updated (diff)

added python migration script which is not needed but at least 10 times faster than grief prevention's. It is ignoring parent ships and non builder trust's, which are barely used) - created as grief prevention didn't want to convert in the first place

The startup time with mysql is the same or even slower on top the shutdown is taking another 6 minutes, now. Therefor we even lost. But it allows easy data cleanup which allowed me to decrease the row count from 36k to 6k as +90% of this change was on the player data table it might not help much though.

B-Team 2 has been migrated, B-Team 1 is going to stay on flat file until we know more.

Actions #9

Updated by Slind over 9 years ago

  • Description updated (diff)

added DB cleanup queries used

DELETE
FROM `griefprevention_playerdata` 
WHERE (accruedblocks = 200 AND bonusblocks = 0) OR lastlogin < NOW() - INTERVAL 14 DAY;

DELETE c
FROM `griefprevention_claimdata` c
LEFT JOIN griefprevention_playerdata p
ON c.owner = p.name
WHERE c.builders = "" AND p.name IS NULL;

Actions #10

Updated by Slind over 9 years ago

  • Estimated time changed from 4.00 h to 5.00 h
  • % Done changed from 0 to 80

B-Team 1 migrated and cleaned up.

Actions #11

Updated by Slind over 9 years ago

  • Resolution set to Fixed/Completed
  • % Done changed from 80 to 100
  • Assignee changed from Developer to Slind
  • Status changed from New to Closed

reduced to around 2m 30s on b-team 3 but shutdown delay increased by around 1 minute. As it is the only old server with this version of griefprevention that is ok to live with.

Actions

Also available in: Atom PDF