Defect/Bug #94
closedGrief Prevention Loading Time at 5 min. - B-Team
100%
Description
The claim loading time on B-Team is extremely high. We should either look for an automated cleanup of claims from players that didn't spent much time or high improve the performance of the loading process.
Comparison:
B-Tema 1 (flat file) loading time 6m 32s claim data 6.421 player data 34.694 B-Team 2 (flat file) loading time 6m 4s claim data 7.571 player data 28.947 B-Team 3 (mysql) loading time 35s claim data 2.388 player data 7.374
Script for flat file to db migration (not needed but at least 10 times faster than grief prevention's, but it is ignoring parent ships and non builder trust's, which are barely used) - created as grief prevention didn't want to convert in the first place
#!/usr/bin/python import MySQLdb import os db = MySQLdb.connect("localhost","user","password","database") cursor = db.cursor() for fn in os.listdir('./plugins/GriefPrevention/ClaimData'): claim_id = fn claim_data = open('./plugins/GriefPrevention/ClaimData/' + fn, 'r') lines = claim_data.readlines() claim_data.close() if len(lines) > 4: claim_lessercorner = lines[0] claim_greatercorner = lines[1] claim_owner = lines[2] claim_builders = lines[3] cursor.execute("""INSERT INTO griefprevention_claimdata (id, owner, lessercorner, greatercorner, builders, containers, accessors, managers, parentid, neverdelete) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""", (claim_id, claim_owner, claim_lessercorner, claim_greatercorner, claim_builders, "", "", "", -1, 0)) db.commit() print (claim_owner) for pn in os.listdir('./plugins/GriefPrevention/PlayerData'): player_name = pn player_data = open('./plugins/GriefPrevention/PlayerData/' + pn, 'r') lines = player_data.readlines() player_data.close() if len(lines) > 4: player_lastlogin = lines[0] player_accruedblocks = lines[1] player_bonusblocks = lines[2] cursor.execute("""INSERT INTO griefprevention_playerdata (name, lastlogin, accruedblocks, bonusblocks, clearonjoin) VALUES(%s, %s, %s, %s, %s)""", (player_name, player_lastlogin, player_accruedblocks, player_bonusblocks, 0)) db.commit() print (player_lastlogin) db.close()
DB cleanup queries
DELETE FROM `griefprevention_playerdata` WHERE (accruedblocks = 200 AND bonusblocks = 0) OR lastlogin < NOW() - INTERVAL 14 DAY; DELETE c FROM `griefprevention_claimdata` c LEFT JOIN griefprevention_playerdata p ON c.owner = p.name WHERE c.builders = "" AND p.name IS NULL;
Updated by Xfel11 over 9 years ago
Which bteam server are we talking about here? bteam 1 and 2 use a flatfile datastore, bteam3 is using a mysql data store. Which of these is the problem?
Updated by Xfel11 over 9 years ago
So it's probably related to the poor data model of the flatfile data store. One flaw I could identify is the way it loads the claims for one world: it actually has to open all claim files once in order to find out if they are in the currently loaded world, and if the world matches, the file is opened again to actually load the claim. It might be possible to optimize this.
Updated by Slind over 9 years ago
Maybe we can port it from the 1.8 build as I never noticed any issues on there.
Updated by Slind over 9 years ago
BTW. It feels to be exponential, but I have no proof.
Updated by Slind over 9 years ago
- Description updated (diff)
Comparison:
B-Tema 1 (flat file) loading time 6m 32s claim data 6.421 player data 34.694 B-Team 2 (flat file) loading time 6m 4s claim data 7.571 player data 28.947 B-Team 3 (mysql) loading time 35s claim data 2388 player data 7374
Updated by Slind over 9 years ago
- Description updated (diff)
added python migration script which is not needed but at least 10 times faster than grief prevention's. It is ignoring parent ships and non builder trust's, which are barely used) - created as grief prevention didn't want to convert in the first place
The startup time with mysql is the same or even slower on top the shutdown is taking another 6 minutes, now. Therefor we even lost. But it allows easy data cleanup which allowed me to decrease the row count from 36k to 6k as +90% of this change was on the player data table it might not help much though.
B-Team 2 has been migrated, B-Team 1 is going to stay on flat file until we know more.
Updated by Slind over 9 years ago
- Description updated (diff)
added DB cleanup queries used
DELETE FROM `griefprevention_playerdata` WHERE (accruedblocks = 200 AND bonusblocks = 0) OR lastlogin < NOW() - INTERVAL 14 DAY; DELETE c FROM `griefprevention_claimdata` c LEFT JOIN griefprevention_playerdata p ON c.owner = p.name WHERE c.builders = "" AND p.name IS NULL;
Updated by Slind over 9 years ago
- Estimated time changed from 4.00 h to 5.00 h
- % Done changed from 0 to 80
B-Team 1 migrated and cleaned up.
Updated by Slind over 9 years ago
- Resolution set to Fixed/Completed
- % Done changed from 80 to 100
- Assignee changed from Developer to Slind
- Status changed from New to Closed
reduced to around 2m 30s on b-team 3 but shutdown delay increased by around 1 minute. As it is the only old server with this version of griefprevention that is ok to live with.