ABOUT: ProjectEQ Expansions Project

ProjectEQ; the Open Source Database for EQEmulator Since (2003-ish)

For a long long time; the EverQuest Emulator community has been basing every EQEmu Open Source server installation off of ProjectEQ's database. This database has been built from a wide variety of reverse-engineering, data collection, analysis efforts and many folks contributing to its "live-like" accuracy over the years

Today; I'm going to be discussing a few topics, but mainly I'm going to be talking about one large initiative and issues surrounding said topic and everything I've observed over the years

Contributing back to ProjectEQ

Lots of folks will use this database; play on it as is, slightly make tweaks or different rulesets or completely make their own flavor and spin off of the amazing game we've all love have been addicted to for a ridiculous amount of time (20+ years). One of the biggest things that has always excited me about this project is people are able to enjoy this amazing game in a different light in whatever fashion that they desire, playing on their own server with family, hosting with a small group a friends, hosting for a large wide audience, or just playing privately on their server with bots

There are also lots of folks who spend a TON of time creating true "classic" or era-specific experience based off of a very modern database dump and never contribute back to the origin project (ProjectEQ)

Contribution Barriers

We've seen so many people come through this community and put in amazing work, only to be enjoyed by a very small audience or not at all because there are large technical barriers that keep or demotivate people from being able to bring fixes back into the ProjectEQ database; some of these barriers may include

• Technical knowledge limitations of how to take what's in their local database and contribute it back to the source database; without conflicting table ID's in many different tables depending on how little or big the changes may be
• The very slow feedback loop of contributing changes back due to technical barriers
• Not having clear contribution guidelines
• Needing to have a slowly gained reputation on ProjectEQ and hard ramp-up to be able to make contributions on ProjectEQ
• Feeling limited by having direct access to ProjectEQ tools to be able to edit, make fixes / adjustments
• etc.

Today ProjectEQ is at Omens of War and has content mostly "filled out" up until Omens of War era, but one thing that is painstakingly obvious that a lot of people notice time and time again is that many things are either wrong, broken or inaccurate. All of these things that folks find, once again they may have fixed on their own servers, time and time again and it never makes it back up to ProjectEQ because they either want to hold their work close to their chest or they simply just don't know how to do so

All of the above mentioned issues and more have kept ProjectEQ back in many ways. We have an amazing staff of dedicated contributors who work tirelessly making our server more and more accurate, but its the same small group of staff we've had for years who have had to earn their way up the trust ladder to gain privileged database access via tools and those who've proven themselves to work tirelessly to contributing to the project, these guys we can never thank enough.

ProjectEQ Expansions Project & Multi-Tenancy (Trusted Database Crowd Sourcing)

As a facilitator of this project; I aim to improve developmental flows, processes, systems, infrastructure to make the lives easier of all of the people in the EQEmulator and ProjectEQ community who are working to contribute to both. With that facilitation in mind; I've had quite a bit of time to think about how we can solve the problems of contributing back and era-accuracy at the same time

Multi-Tenancy

What is multi-tenancy?

In the enterprise software world; this typically means that your application is able to simply store data that is meant for different customers (tenants), to live in different logical databases. In our case; it is a means to "fan-out" the source of our database connections from the server level

If we can have the ability to have our emulator server be able to have its own local player, system, state tables and be able to use an external content database then we can achieve a level of isolation that allows us to CDN or (crowd-source) our already publicly available data in a "live-edit" fashion

This would enable us to accomplish:

Focused Crowd Sourcing Database "Live-Edits" (CDN)

Hand out database connection users to trusted or very newly interested people who have a desire to contribute towards the project without requiring direct tool access, direct GM access to ProjectEQ and they can run the content database tables over the internet to their local development server with relative ease just by adding a few lines in their server configuration

This would allow people to work on parts of the server using their isolated local server; write new quests based off of newly adjusted NPC data, newly adjusted spawn data, trade-skill data etc. that is "live-edit" published to the ProjectEQ content database. When a developer is done making their changes on their local, the database changes will already be published and they can submit a pull-request for their quest changes to the ProjectEQ quests repository when they are ready to push them

We've considered a UUID approach for all of our tables, which would be massive in scope changing schema over many dozens of tables and adjusting source code. This would also require extensive import / export tooling and a lot of edge issues that we simply do not have the time for in a volunteer-based project. Going this route makes a LOT more sense for contribution

Expansions Project

Stand up multiple ProjectEQ servers that represent each era; all using the same content database tables.

We would make architectural changes in all of the necessary content tables to have a context of expansion and content flags which simply will only return era-specific data back based on the local settings of the server

With multiple "test servers" online representing each era; we can then work on making the same exact database true to each era individually and separately all at the same time.

When people install a new local server; they will have more control over what era they want their server installation to reflect without a massive undertaking by being able to flip a "content flag" or "era flag"

ProjectEQ Test Servers

As mentioned before; as this evolves, it makes sense for the team to stand-up servers that represent a specific era to maintain quick testing, feedback loop. Each server would have its own isolated character database, system settings etc.

Having these servers up as playable servers also allows for folks to play during this era and help the project push closer to making it more and more accurate

​

Trusting Contributors

Our architecture of "CDN-ing" our content database server and allowing developers to "live-edit" against the same database that all the server are using would require a few things

In the architecture above; we have ProxySQL in front of the server connection; which would:

• Enforce ACL's for malicious activity
• Log all SQL statements that write to the content database
• Live stream database writes to a developer channel so it can be monitored
• Allow per-user writing privileges to be tweaked as needed

Today; we've never had any issues with the above, but having these things transparently available to the team keeps everyone honest

Backups

The ProjectEQ database gets backed up rigorously, so in case a worst-case scenario is evident, recovery plans are fairly straightforward. We also have a publicly available open-source ProjectEQ database archive available here for content data already: http://db.projecteq.net/

Timeline

The multi-tenancy work is almost complete (https://github.com/EQEmu/Server/compare/feature/multi-tenancy), with a handful of cross-boundary or cross-tenant queries that need to be addressed and that portion of the work will be complete. I will be sliding into the expansions server side work very shortly; looking to have both available as soon as possible so we can get this project off the ground

Q & A

Q: Can I contribute to ProjectEQ in a specific-era?
A: If you have aspirations to contribute to a specific era because its what you're interested in, then standby until resources become available and the core team will evaluate outside help on an as-needed basis. In general, ProjectEQ team is open to contributions from everyone as long as they meet guidelines

Q: Does the expansions project require multi-tenancy?
A: No, you do not need multi-tenancy for this project; it will just make it far easier to build, maintain and make it even feasible to develop from the perspective of ProjectEQ

Q: Are you trying to compete with Project1999?
A: Not at all, we feel Project1999 has the classic experience on lock down to the most detailed sense, our project serves a similar but different purpose in the sense that its what the entire community has used as their server base for over a decade and our focus has been more modern content, but wishes to backfill era accuracy as much as we can

Q: Do we have to use the CDN connection to contribute?
A: Not at all, people can still contribute quests and changes without having a connection; although certain things may be made more difficult. Lots of people have been making contributions over the years without any of these solutions available

Conclusion

This thread is up for Q & A; as folks ask more questions; I'll do my best to answer them. I anticipate even members of the ProjectEQ team will have questions and want to discuss logistics as well as this develops

 

Update

I'll be entering the next phase of this fairly soon; I have an integration branch containing the repository work, expansions work and multi tenancy work from last week

I've setup a content database CDN deployment with proxy configurations all ready to go

JJ is working on making changes to the PEQ editor to allow for the multi-tenancy split

I'll be splitting up ProjectEQ's Live database afte we've done a few days of testing using the test server and the content DB CDN

I have some small finishing adjustments to make WRT to expansion work, but it is 80-90% of the way there which is where I want to checkpoint the work before we continue moving forward with additional changes

https://github.com/EQEmu/Server/compare/integration/multi-tenancy-expansions-repository

I'm back at work this week so when I pull the trigger on the testing phase is to be determined

 

Glad you're excited

We're getting close to the content development phase of everything so it will still be a while before we open up any public facing servers specifically; but it could be possible we do a progression oriented structure just so we can get help from players to help critique things that need fixing in the era as we go along