Monsters & Memories - An Old School Indie MMO

[Swish]

bump, loving how this is going.

[Topgunben]

Originally Posted by Swish [You must be logged in to view images. Log in or Register.]

bump, loving how this is going.

Sadly its probably the only game I look forward to.

[Trexller]

Originally Posted by Topgunben [You must be logged in to view images. Log in or Register.]

Sadly its probably the only game I look forward to.

same here

maybe we should give them a big pile of money

[Zukan]

Hey guys here's some of the take aways from the internal friends and family stress tests we did over the weekend, as we prepare for the public alpha which will hopefully be a lot more people. [You must be logged in to view images. Log in or Register.] Theres a bunch of techno-bable that goes over my head but the short of it is our tests were really successful! Things held up, and the stuff that had some issues we've got solutions or already have resolved.

Hey all! Over the weekend we held two Stress Tests with our Friends and Family tester community. Ali wrote up a very nice breakdown in our Slack. There was no overly sensitive information, so I thought it'd be fun to share it with you all! Enjoy. (Apologies for the incoming wall of text crit!)
[8:37 AM]
Hey y'all! So some interesting issues, observations and metrics from our load tests.

We got around 80 clients both days.

I will break down these issues into their own threads below.

On the first day these issues were noticed:
1. The immediately noticeable micro stutter that happens approx every 20 seconds. The timing made sense as we have an expensive Core.Models.ClientModel.Save() happening on that timer as well.

2. When massive AoE hits clients, it causes the whole server to stop while it processes what happened.

3. Players that are far away and are supposed to exit your sphere of influence sometimes hung around indefinitely.

On the second day these issues were noticed:
1. There was visible latency on positional updates when a lot of players were close around you, but this improved as you moved away. This was consistent with my mass bot tests last year and we have logic that needs to be fine tuned to counter act this.

2. Someone had made a corpse art piece at the entrance. It was great! Showed us that corpses add unreasonable load.

3. The errors: 7B2EE78E and 6D24DB45 were observed.

General Issues unrelated to Network Test:
1. The model fail errors were noticeable (nameplate visible but invisible model). These may be solved with the new texture and attachment systems but will have to be tested when FnF is updated to latest master.

2. Group member UI bugs. When you zone or die the entire group UI becomes unusable.

Observations:
- As expected, NPCs add a lot more load on the server than Clients. There was virtually no increase in load between <10 players and >70 players.

- The network library held up really well! I always had a nagging fear that with more clients we would see packet queue issues. However, there were none that I could see.

- That said, we need better network profiling tools!

- There is a lot of optimization required in client saves and positional updates especially when we have a lot of users in one area.

Metrics:
- Night harbor (our heaviest zone BY FAR) runs at 2.4% memory required (~400MB), over 41 threads, and uses up CPU load 75% (out of 800% max) @ approx 3GHz (per core, 8 cores on the server). This is at full load with multiple dragons. When there wasn't much going on it was hovering at around 55%. The main driver of this load is navigation and the very large navmesh that NH has.

- Save stuttering was noticed when we got to over 40 clients.

- Positional update stuttering is noticed when more than 40 clients and tons of corpses were at the gate, and you are within 100m of the gate.

Overall:
It was a great success! I think for our first load test, this was amazing and we got a lot of good metrics and I think I have a clear plan for us to solve the above.

Considering nothing crashed, nothing ran out of memory, the zone did not break in any way, and all the issues we found are things fixed with minor optimizations, we are in a really good place. I look forward to fixing them and then testing with hundreds in the same zone!

Microstutters:
I have already solved this. I have converted the save algorithm to a scheduled queue, where we queue the saves on a timer then handle saving over multiple frames and use transactions as much as possible.

Further optimizations: All client saves should be done via queue, even for things that require immediate save, unless we specifically say we need this INSTANTLY (such as trades or taking a port).

We need to have live database performance profiling that we can pull from a running server (maybe by enabling it with a gm command, then extracting the results and also disabling it).
[8:37 AM]
AoE:
We need to look into this one more. There are two things at play here:

1. The client stutters anyways when a lot of VFX is spawned at the same time. We can likely queue these and spawn them over 4-10 frames without it being noticeable at all to the user.

2. I think when the damage, deaths, and buffs are applied, they all immediately save, causing an 80 client mass save in one go destroying our database pipe lol.

Players not exiting SOI:
We need to audit the code that defines this. Maybe there's a logic issue there.

Positional update delays when when many entities (and especially when many clients) exist within a SOI:

This is in effect not too difficult to solve, but explaining it will be tough because it also relies on knowledge of both our SOI system and other techniques that we can move to.

The way SOI works right now for us is that we check all entities against all other entities to see if we need to enter or exit them from any SOI. The logic goes:

for entity a of entities:
for entity b of entities:

var shouldBeInSOI = b.shouldBeInSOIof(a)

if (not in SOI and shouldBeInSOI) then run entryCallbacks
if (inSOI and not shouldBeInSOI) then run exitCallbacks

Then separately we run:

var positionUpdates
for client a of clients:
for entity b of a.sphereofinfluence:
positionUpdates.Append(b.position)
a.send(positionUpdates)

This has a couple of pain points:
1. Building packets is relatively expensive.
2. This type of packet accounts for over 90% of all our traffic.
3. It is exponentially growing load with more clients within each other's SOI.
4. Each client must build its own positionUpdates.

Possible solution:
Client's SOI has many variables (such as being a pet of client, or a party member, or in combat) but a big portion of what defines SOI is distance. This can be substituted for a graph distance. Especially useful would be an octtree, but I see that it may be simpler and ultimately easier to implement a regular 3D grid.

So instead of checking if an entity is X meters away from another entity, we check if the GRIDPOSITION is within X GRIDDISTANCE from another entity.

What this ultimately allows us to do during building packets is to build a positional update PER GRIDPOSITION and then append the relevant ones by directly serializing them like we are doing currently for positionUpdates (skipping the garbage collection), and then finally appending any additional required out of distance SOI entities.

This would be much, much faster in my estimation by orders of magnitude, since we don't need to do positional reads per entity squared.

We may also find that its more suitable to send this all in two packets, one that is faster using only grid position, and another that is slower for in SOI but out of grid distance.

Removal of existing optimizations:
We currently have an optimization where if we exceed a certain number of clients within SOI, we start throttling update packets. This is I believe the main thing being observed with the slower interpolations and especially when clients are jumping around.

Other factors:
Implementing parabolic trajectory code and using that for jumping will smooth out the noticed clipping into the ground when clients are jumping around in high latency or throttled position update situations.

We should also let the client know and allow the client to adjust its expectation of the frequency of updates for smoother interpolation. Currently, client is still interpolating using its original assumptions which can cause jerkiness.

[Zukan]

Details on Public Test released this morning, here is aLovingRobot's post from our Discord

Hi Everyone!

There have been a lot of questions about our first Open/Public Stress Test.

We originally targeted the end of this month (March 2023), but have decided to change that based on details that will be explained below.

The following plan is now firm:

Public Stress Test
April 28-29th (Fri & Sat)

This will consist of two 4-hour sessions (one each day), with the possibility for them to run longer than 4-hours if conditions allow.

The focus of these two sessions is Technical Stability at Scale.

This includes everything from download and installation, through in-game/network/server performance.

Data derived from this test will enable future tests that focus on gameplay in general or other specific facets of the game.

This stress test allows us to gauge where we're at in development and plan for future sessions.


The reasons for the change in timing are as follows:

A recent Friends & Family Stress Test provided us with data on optimizations and bugs that needed to be fixed.

Plus, this places the test right after the end of Ramadan.

The new date gives us the next 3 weeks to wrap up the work on these issues, and then 2 weeks of time dedicated purely to testing and prepping the test build.

As a side benefit, it also gives us more time to get in the new character art and address a number of other art and gameplay issues/additions, that while not the focus of this test, will make your first impression that much better.

We'll update the information provided here in the coming days to include specific times and any details we might have left out.

We're currently evaluating the best window of time for our North American folks, since you/they make up the vast majority of our community and team.

We're looking forward to seeing you on the Friday, April 28th and Saturday, April 29th.

Thanks!

[Zukan]

608 people rsvp'd so far. [You must be logged in to view images. Log in or Register.]

[mrgoochio]

Awesome news

[magnetaress]

Still looking forward to this! Thx for the updates!

[Zukan]

March update just dropped! I recommend you check it on the site here because there more images and videoes than I will be posting here and this month has a bunch of cool character stuff so check it out.

[You must be logged in to view images. Log in or Register.]

Hi all!

We said last month that we were hoping to have a Public Stress Test by the end of March.

After running through the feedback and issues from a recent Friends & Family Stress Test, we’ve decided to push the public test until the end of April (28-29th).

We’ve posted more information about this in our Discord, so we hope you take the time to swing by and read the rest of the details.

[You must be logged in to view images. Log in or Register.]

Again, check our Discord for more info on the upcoming stress test.

As I type this, over 800 people have expressed their interest in trying to smash our servers that weekend, so… well… it may end up being chaotic, but it’ll be a fun milestone either way.

And until then, here’s some data we’ve pulled from our Friends & Family Testing.

[You must be logged in to view images. Log in or Register.]

And so… while the upcoming Stress Test has been a big focus for design and tech, we’ve also got plenty of other work done, and a ton of new art to look at. Enjoy!

Character Art & Concepts
This month we completely rebuilt the character texture loading system. Ali revamped the tech and the art team prepared new human male model and textures, and also transitioned a bunch of old textures to the new system.

Also:

• We can tint everything with color gradients now!
• New faces, facial hair, and armor sets added
• Hair and facial hair still a work in progress
• Updated textures to match the new UV layout
• Created a new shader for the fire elementals
• Reworked Ghost shader to work with both the new lerped gradient texture system, and diffuse textures
• Human Male nipples added (Happy Birthday, Shawn!)
• Saber Model added
• Dragon Shield added
• Backpacks can now change colors
• Pouches can now change colors

[You must be logged in to view images. Log in or Register.]

A note from Goblin regarding the Concepts Above:

“I wanted to include this to give context to our new armor system and how it affects concept art.

Each gear slot has a maximum of 3 different colors we can apply to the piece of gear. It's enjoyable for me to problem solve what 3 colors gets allocated to what gear slot, but it also pushes some nice repetition, and complexity to the designs.

Take for example the torso slot. It overlaps the shirt slot, so we essentially have 6 possible overlapping colors on that one area. This only gets more complex when mentioning shoulders and capes, which can share the shawl space.

Zukan and Patrick (vgfx) were incredibly prolific in their desire to simplify the system we had - and this revealed a manageable complexity to our armor system, while also offering more novelty in the designs.”

[You must be logged in to view images. Log in or Register.]

Audio & Animation

• Ocean sounds tweaked
• Floating Dome sounds added
• Shaded Dunes music and sound effect setup
• Merchants now have music
• Crocodile movement sound fixed
• Rat and Ashira footsteps quietened
• Added new human animations
• Idle animations for Ghoul, Wererat and Ogre

[You must be logged in to view images. Log in or Register.]

Environment Art & Concepts
Night Harbor

• Northern Newbie Yard revamp gray boxed complete
• Sage Side continued updates
• Night Harbor Nav Mesh Optimization
• Various geo fixes / adjustments across the zone
• Rough lights added through Night Harbor to help with night time navigation
• Fishing Volumes Added
• Exterior block-out of Mirthful Djinn (Inn)
• First pass on the "Elemonkalist" Guildhall and adjacent cavern
• Collaborated on first pass of Yuum & Gyffre statues with Goblin
• Completed interiors for the majority of Twilight Harbor buildings
• Modelled props based on concept art for the Elementalist Guildhall

Wyrmsbane

• Fixed invalid vert issue causing black screens in the antechamber wing
• Fixed light triggers in Elite tomb area
• Fixed an issue causing max shadow count to be reached in certain rooms, causing shadow flickering

[You must be logged in to view images. Log in or Register.]


Game Design
Content

• Northern Newbieyard has been updated
• Rogue starter quests have been added
• Completed initial quest writeups for the rest of the class guilds
• Began implementation of the remaining first stage newbie quests
• Various Night Harbor population adjustments
• New Zone MUD interactions
• Began deity lore expansion

Itemization

• Tattered Cloth Robes can now be broken into scrap
• Tattered Cloth Robes can now be upgraded to Patched
• Patched Cloth Robes should work properly now
• Fine Cloth items that wouldn’t sell should now sell to Cloth merchants
• Cheikho in the hunter camp now returns home after combat
• Paladin Merchant no longer sells dispel and abolish magic
• Bronze Bar added
• Copper Bar trivializes at 25 Smelting
• Rusty Bronze renamed Corroded Bronze

Mob Abilities

• Snake Kick is no longer also an Interrupt

General Abilities

• Skeleton Pet damage reduced by 25%
• Added Conflagrate, Flames of Eth, Fires of Eth
• Added Life Duct, Life Flume, and Exsanguinate
• Various Abilities now have higher mana costs and cast times

Archer

• Auto Fire now shares a cooldown with Auto Attack
• Targeted Shot is now Instant Cast

Beastmaster

• Rat Pet melee damage reduced by 25%

Druid

• Added Flames of Eth-ehm, Fires of Eth-ehm, Embers of Eth-ehm

Elementalist

• Fire and Water Pets now have 40% less HP
• Air Pets now have 10% less HP

Rogue

• Swift Strikes can no longer miss and is now self targeted

Shaman

• Fixed an issue with several Shaman Slow spells that were not properly applying their debuff

[You must be logged in to view images. Log in or Register.]

Tech

• Completely new Character Controller
• Completely new model loading process
• Ranged weapons now have models and are visible when you use ranged attacks
• New faces for Human Male integrated
• Positional update and Sphere of Influence fixes to support more players in close proximity
• Added command /chatfontsize which configures the chat font size
• Fix for using /place <object> with an item in the cursor
• Follow-up fix for cross-zone GM summons, actually fixed this time
• Corpse model loading is now rate limited, to prioritize other loading entities first when there are many corpses around
• Support for teleport volume sound
• Improved group pet frame UI functionality. Added support for multiple pet frames, and auto-layout
• Code architecture improvements
• Fixed some issues with party invite / rejection logic and UI
• Fixed some issues with party frames when members are in different zones
• Added resizable chat window functionality
• Added better support for user settings with dropdown UIs
• Added right-click option to reset a single UI element
• Fixed an issue with the chat window in large resolutions

Fixes

• Bards can skillup their music skills again
• Pet AI won’t error out if its master is missing
• Fix targeting issue on charm dropping
• Fix charm pet aggro on charm fade
• NPC AI awareness of buff overwrites, will not attempt to cast a buff that will not stick
• Zoning in while pet is already summoned no longer breaks UI
• Can no longer bug mobs on corners
• Party fixes
• Fix party frames for players in other zones not showing player name
• Corpses should no longer clog up the model loading pipeline
• Teleport sound triggering adjustments

Changes

• Skill gain rates adjusted
• Tradeskill success chance rates adjusted
• Trivial combines will let you know they are trivial
• Add a cooldown between melee auto attacks and ranged auto attacks
• More advanced pet bars, auto arranging group UI with pet bars, UI support for multiple pets
• Add ability name to cast bar
• Client saving optimizations
• Music states for better audio immersion

[You must be logged in to view images. Log in or Register.]

In Closing
Last but not least:

We’ve had the pleasure of adding another member to the team again this month!

Brandon (“First” in Discord/game) will be supporting us on the design side of the house. First’s contributions during Friends & Family testing and subsequent discussions we had on topics like worldbuilding, lore, and design in general, made it a no brainer to ask if he’d be interested in working with us.

We’ll get a bio up for him on the Team page soon.

~~~

Obviously, we’re going to be very focused on preparing for the stress test at the end of the month.

You can follow that progress via our new Twitch category and the VODs we post on YouTube.

Join the Discord and Mailing List if you haven’t already.

~~~

A massive thanks for being a part of the community!

[Topgunben]

Damnit these guys are doing a great job. Can’t wait to play.