Porting iniDB to a more generic stats saving (sock-rpc-stats)

  • Offline micovery
  • Moderator
  • Hardened
  • ******
  • Posts: 159
Hey I am the author of the sock-rpc-stats Node.js module, and sock.dll/sock.so extension for Arma 3.

sock-rpc-stats is a Stats Server abstraction for Arma 3 that lets you add persistence to your mission in a database-agnostic kind of way.

You can read more info about it here:

http://forums.bistudio.com/showthread.php?183892-Stats-Server-and-Client-API-with-support-for-SQL-and-NoSQL-databases

Here is a highlight of the features:

0. Works in Linux, and Windows
1. Support for multiple storage systems/databases (File system, MySQL, MongoDB, Redis, CouchDB, Cassandra)
2. No limit in length of data you are saving, or retrieving from the storage system (works around the callExtension buffer size)
3. SQF data types are mapped to JSON, and stored as JSON (mapped back to SQF when retrieved)
4. Automatic stats backup
5. Simple to use  set/get API that works from both client-side, and server side.



Anyways, so looking at the github repos under:

https://github.com/A3Wasteland/ArmA3_Wasteland.Altis
https://github.com/A3Wasteland/ArmA3_Wasteland.Stratis

I see that right now they are using iniDB, I am wondering if there is any interest in porting or maybe adding side-by-side support for this stats saving system.

I am willing to fork the missions, and spend some time to port the current iniDB system, if people are interested.



  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #1 posted: Oct 09, 2014, 06:43 AM »
Not sure if you might be late to the party? Torndeco is almost done porting us to extDB. Although, I'd imagine you could add in support for your callextention.

http://forums.a3wasteland.com/index.php?topic=385.msg3393#msg3393

TOPArma Mission - Share your mission on GitHub
  • Offline micovery
  • Moderator
  • Hardened
  • ******
  • Posts: 159

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #2 posted: Oct 09, 2014, 07:25 AM »
Not sure if you might be late to the party? Torndeco is almost done porting us to extDB. Although, I'd imagine you could add in support for your callextention.

http://forums.a3wasteland.com/index.php?topic=385.msg3393#msg3393

The more reason to see if people are interested :-) ... if everyone is happy with iniDB, or extDB, then there is no point in spending the time to port it.

Though looking through the source code, looks like the interactions with the database could be generalized into these operations:

get  (get stats)
set  (set stats)
delete (delete stats)
init  (initialize the database)

In theory there could be a layer of abstraction ... that would expose those operations, and people could just hook them to the whatever stats system they wanted.

At the moment all the calls seem to be tightly coupled with the "iniDB"  API.

Or not ... I see PDB_write, and PDB_read, and others that look like some generalization switching between iniDB or profileNamespace ...,so I could technically add support using the same mechanism.

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #3 posted: Oct 11, 2014, 11:16 AM »
I guess anything that have less(er) impact on server performance is good. Many of us have different wishes on how saves should work and how easy it would be for a server operator to access that data and how it is presented.

Many thanks for helping out!
  • Offline micovery
  • Moderator
  • Hardened
  • ******
  • Posts: 159

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #4 posted: Oct 12, 2014, 10:02 AM »
I guess anything that have less(er) impact on server performance is good. Many of us have different wishes on how saves should work and how easy it would be for a server operator to access that data and how it is presented.

Many thanks for helping out!

Makes sense.

I have already ported it with minimally invasive changes. Basically, I created wrapper for all iniDB_xyz functions, and made them call my sock-rpc-stats library instead.

Here is a video of me setting up the Wasteland Altis mission using the sock-rpc-stats library, and couchDB as the storage system in Linux:




The instructions from the video are here:

https://github.com/micovery/Release_Files/tree/master/sock-rpc-stats

Note that I chose to use couchDB in the video, just a proof of concept, but you could technically set it up to use any of the supported storage systems (MySQL included).

The forked source code with the sock-rpc-stats support is at:

https://github.com/micovery/ArmA3_Wasteland.Altis


On the topic of performance, I noticed a couple of design issues with the mission:
  • The player stats are saved on a loop ... instead of during player disconnect
  • The player stats are retrieved one by one (when player connects), instead of as a batch.

  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2652

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #5 posted: Oct 12, 2014, 11:26 AM »
I don't want to downplay micovery's effort, you've done quite a nice job, but Torndeco is currently working on a MySQL version, using his own extDB addon, which is quickly becoming the de facto standard for cross-platform MySQL integration in Arma 3.

He's got an early but surely working build here: https://github.com/Torndeco/ArmA3_Wasteland.Altis
  • Offline micovery
  • Moderator
  • Hardened
  • ******
  • Posts: 159

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #6 posted: Oct 12, 2014, 12:17 PM »
I don't want to downplay micovery's effort, you've done quite a nice job, but Torndeco is currently working on a MySQL version, using his own extDB addon, which is quickly becoming the de facto standard for cross-platform MySQL integration in Arma 3.

He's got an early but surely working build here: https://github.com/Torndeco/ArmA3_Wasteland.Altis

Hey AgentRev, thanks for the official reply.

Looking at Torndeco's fork ... I see that he has made a lot of changes throughout the source, and not just simply plugged in extDB. 


A few things to consider with the extDB port:
1. The mission is going to be hard-bound to the MySQL schema (as defined by Torndeco)
2. The mission is going to be tightly coupled with extDB, with low level calls sprinkled throughout
3. Maintaining a SQL schema is not easy ... indexes, primary keys, table stats, etc
4. Migrating between versions of the mission, when schema changes ... not a fun task for administrators.


sock-rpc-stats, just like iniDB is schema-less, hence it was so easy to port the mission by simply overwriting the iniDB_read, iniDB_write, iniDB_exists, and iniDB_deleteSection functions.

My view is that persistence for Arma 3 missions should be second-thought, and simple to use (iniDB is great at that). Mission developers should be focused on fixing the things that matter ... like performance, and mission quality.

Anyways, I gave it a try :-)
  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2652

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #7 posted: Oct 12, 2014, 12:31 PM »
The most popular mission in Arma 3, Altis Life, uses extDB. Sure, maintaining a MySQL mission is no easy task, but everybody only has MySQL at the tip of their tongue when you mention persistence, and nobody seems to want anything else but that.

Maintaining a MySQL schema is super easy with MySQL Workbench, but indeed doing modifications to a database that is already in place can be a real mess. But since player data is not that much important as Altis Life, and that server wipes are very commonplace in Wasteland, I guess most players are able to accept their data being wiped to update the database.

I've got no problem with the mission being tightly-bound to extDB, honestly.

Other than that, profileNamespace saving works directly out of the box and is extremely simple to use; it's only downside is that all the data is locked away in a serialized file, and the only way to delete anything externally is to delete the file itself.
  • Offline micovery
  • Moderator
  • Hardened
  • ******
  • Posts: 159

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #8 posted: Oct 12, 2014, 01:06 PM »
You probably already made up your mind ... so this is more of an academic discussion :-).

Indeed, designing a schema with MySQL workbench is a breeze (when it does not crash) ... though shoehorning  a schema where one is not needed is an overkill.

Yep, MySQL is what every one talks about for persistence these days for missions in the Arma community. May be because they have not been exposed to enterprise NoSQL databases like MongoDB, Cassandra, and CouchDB.

sock-rpc-stats does support MySQL ... but the data-model is still schema-less. The JSON data is flattened, and stored as key-value pairs transparently.

Finally, for performance, sock-rpc-stats supports batch read, and batch write ... you could read and write as many keys-value pairs as you want with a single callExtension call. However, for my port I chose to keep the existing design that you have with iniDB where the key-value pairs are read one at a time ... in order to keep the mission changes at a minimum.

  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2652

Re: Porting iniDB to a more generic stats saving (sock-rpc-stats)

« Reply #9 posted: Oct 12, 2014, 01:56 PM »
when it does not crash

LOL so true...

Indeed about databases, I have a comp sci degree, yet all we were exposed to in college was traditional crap like SQL Server and Oracle DB (aka mother of all turds), and all my jobs so far have been in the government, which uses these DBs too. The only NoSQL thing we saw in college was the Google App Engine Python DB Datastore (which was a true godsend) used with an Android app. I really wish we had seen more cutting edge stuff like that. All we did was C#, C++, VB, PHP, and *shudders* Java, you know, all the classic stuff  :P

I live a city which houses a lot of govt departments, so unfortunately most of the IT industry around here is all about boring obsolete government money sink projects, which might explain what we're being taught in class.