extDB development and more

  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

extDB development and more

« posted: Aug 07, 2014, 12:41 AM »
We are finding limitations to adding more servers in our cliche for persistent across all servers. We are seeking for developers to provide a bid to topwasteland@gmail.com or here.The bid will be for all items completed below unless specified otherwise. Please provide a timeline of completion and cost of project.


Implement extDB as an option for saving, directly to mysql.
Provide the ability to have multiple servers interface with one database (mysql).


Once completed, the code would be released to AgentRev for option to include into the main release. We hope this will help alleviate some of the work load upon AgentRev. We are interested in furthering the development of A3Wasteland.

TOPArma Mission - Share your mission on GitHub
  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2600

Re: extDB development and more

« Reply #1 posted: Aug 07, 2014, 01:52 AM »
Now that is a good idea  ;D

If anybody is up to the task, here is a starting point:

https://github.com/Panaetius/ZCG_A3Wasteland_Altis/tree/master/DatabaseSetup
https://github.com/Panaetius/ZCG_A3Wasteland_Altis/tree/master/persistence

This is a (semi) working MySQL implementation, however it has 2 major faults: it uses Arma2Net, which doesn't run natively on Linux, and it is limited by the 4096-char buffer from callExtension, so all inventory-related fields would ideally need to be selected/updated individually instead of in a single call (especially magazine ones).

The 3 main types that would need to be split into different functions are: players, objects, and vehicles
So, all the "basepart", "beacon", "warchest", etc. functions would be need to be regrouped as a single "object" type.

All code should ideally be well aerated and written in Allman style. (e.g. like this instead of that)
  • Offline JoSchaap
  • Developer
  • Mercenary
  • ******
  • Posts: 483
  • Had a life.. Got a modem.. (~1996)

Re: extDB development and more

« Reply #2 posted: Aug 08, 2014, 09:59 AM »
hmm they said it the biki in future versions it can be increased..

cant find a ticket asking them to increase it though :)


https://community.bistudio.com/wiki/Extensions#DLL_Interface

Quote
The game currently calls the RVExtension function with outputSize of 4096 (can be increased in future versions if needed, called extensions should always check this value and never produce output larger than this size).


looking into extDB (and i might be wrong here) it seems it supports multipart messaging of results. (possibly to by pass the 4096 limit?)

https://github.com/Torndeco/extdb/wiki/Calls:-General-Info#retrieving-async--save-message--multi-part-message

then again, i could be mis-reading the concept (it's still early here)

Re: extDB development and more

« Reply #3 posted: Aug 08, 2014, 02:55 PM »
Basicly extensions should be made so the buffersize value is not hardcoded into it.

For example for awhile there Linux (had older smallersize) + Windows Servers (had newer larger size) different buffersize for RVExtension
Not sure if thats still the case, cant be bother to check...

-----------------
-----------------

Anyway extDB works around the buffer limit.

1) By splitting up the return string.
2) There is also a unique_id for results, so there is no worries about mixing up return values.

-----

Basicly when u call extDB (ASync) it returns a unique ID.

You call the extension again using the Unique ID
Extension will either return Wait (i.e query not done yet) or Result (part 1)

You keep calling the extension using the same Unique ID until u receive an empty string "".
You just add all the parts together to get the full result.

Examples of some code (for ASync Calls)
https://github.com/TAWTonic/Altis-Life/blob/master/extDB-Build/life_server/Functions/MySQL/fn_asyncCall.sqf (db locks)
http://pastebin.com/MXFgAzxC (no locks, not tested really)

----------------
----------------

Note there is a test application the .exe.
Its basically the extension compiled as executable u can use to test sending / receiving values.
  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2600

Re: extDB development and more

« Reply #4 posted: Aug 08, 2014, 04:47 PM »
Anyway extDB works around the buffer limit.

That's good to know  :)

I don't really understand why BIS just didn't modify the export or make another one with something like:

Code: [Select]
RVExtension(char **output, int &outputSize, const char *function)
That way, you could specify the buffer size by changing "outputSize" to your own value, and setting "output" to your own char array pointer. This would technically allow any output size.
  • Offline JoSchaap
  • Developer
  • Mercenary
  • ******
  • Posts: 483
  • Had a life.. Got a modem.. (~1996)

Re: extDB development and more

« Reply #5 posted: Aug 09, 2014, 06:50 PM »
fbticket time again? :)
  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

Re: extDB development and more

« Reply #6 posted: Aug 12, 2014, 02:03 PM »
I appreciate all of the information provided here. We are still accepting development bids to further this project.

We have three servers interfacing with iniDB sharing all of the player data files, while each server has their own Objects and Vehicle files. It would be interesting to see what the access time is with extDB vs iniDB.


TOPArma Mission - Share your mission on GitHub
  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

Re: extDB development and more

« Reply #7 posted: Aug 15, 2014, 12:51 AM »
On a scale of 1 - 10 how difficult would this be for a developer?

I think I am going to try to make this happen. If no one has an issue with it, I'd like to continue to update this thread with my progress and seek help if I get into a bind.

Anything that might help me setup for this?

TOPArma Mission - Share your mission on GitHub

Re: extDB development and more

« Reply #8 posted: Aug 29, 2014, 10:01 AM »
Has anything come of this? I am new here so please take it easy!

I think you can work around the buffer by assigning identifiers instead of passing string arrays all over the show. This would of-course come at a minor cpu overhead due to the lookups. Developing a system which saves player gear and stuff across servers should not bee too difficult to achieve. I found an RVExtension which supports JAVA calls so from this you can pretty much do anything with it if it works. Nways I signed up to follow this, I can also pitch in if you guys need some help, but I am clueless on the SQF scripting, but I have a lot of experience in JAVA Enterprise edition.
  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

Re: extDB development and more

« Reply #9 posted: Aug 29, 2014, 03:49 PM »
Has anything come of this? I am new here so please take it easy!

I think you can work around the buffer by assigning identifiers instead of passing string arrays all over the show. This would of-course come at a minor cpu overhead due to the lookups. Developing a system which saves player gear and stuff across servers should not bee too difficult to achieve. I found an RVExtension which supports JAVA calls so from this you can pretty much do anything with it if it works. Nways I signed up to follow this, I can also pitch in if you guys need some help, but I am clueless on the SQF scripting, but I have a lot of experience in JAVA Enterprise edition.

Nothing has come of it yet.

http://forums.bistudio.com/showthread.php?182146-A3Wasteland-and-ExtDB-development

TOPArma Mission - Share your mission on GitHub

Re: extDB development and more

« Reply #10 posted: Sep 01, 2014, 08:16 AM »
I managed to get RVExtension installed yesterday which allows for JAVA integration. Now the plan is to implement a lightweight http client which then persists all player data to another server via web services which will also have a control panel of sorts which you can log into and do some stuff. Now my next question is how often is a save required, or will save on Escape be ok? Also are the Wasteland servers running on a windows machine or linux, because the RVExtension appears to require the .net Visual C++ lib. Lastly what latency is expected for this to not cause issues with ARMA 3, IE what should my utmost worst time be for a persist? Is 50 millis ok?
  • Offline AgentRev
  • Developer
  • Veteran
  • ******
  • Posts: 2600

Re: extDB development and more

« Reply #11 posted: Sep 01, 2014, 08:47 AM »
We don't need a new extension coded from scratch, we need integration of the existing extDB extension into the mission to save data to a MySQL database. This requires mostly SQF and SQL code, not C++ or Java.

Re: extDB development and more

« Reply #12 posted: Sep 01, 2014, 10:27 AM »
Ok I will start at adding in a custom extension written in c++, but I am keeping the webservices and seperating the database and administration panel onto a different server, because If more than one servers data needs to be stored and retrieved some caching will be involved. Lets say we have 3 servers running at 85 people each, this will already be quite a considerable amount of traffic, but I have never tried running 300 odd players on an extDB so I am also not in the best position to comment on this as I believe the data access would directly depend on the database and modder. Even if this doesn't get published or pushed out it will still be interesting to know if I can get it right without any prior game development knowledge which is pretty much driving this atm.

Re: extDB development and more

« Reply #13 posted: Sep 01, 2014, 11:41 AM »
To give you an idea - the current sqf code using the inidb-addon can be found here:
https://github.com/A3Wasteland/ArmA3_Wasteland.Altis/tree/revive_beta/persistence

this code is used for the persistence features in the current mission.


this code should be adapted to use the extdb extension provided by torndeco:
https://github.com/Torndeco/extdb


If this extension is used mysql, sqlite, ... can be used as the backend-database.  it should be a performance improvement over the flat file reads used at the moment.

and it would allow new functionality to be implemented (e.g. leaderboard with players with most money, ...)
  • Offline lodac
  • Hardened
  • ****
  • Posts: 220
  • TOP Arma Server Admin

Re: extDB development and more

« Reply #14 posted: Sep 01, 2014, 06:01 PM »
Ok I will start at adding in a custom extension written in c++, but I am keeping the webservices and seperating the database and administration panel onto a different server, because If more than one servers data needs to be stored and retrieved some caching will be involved. Lets say we have 3 servers running at 85 people each, this will already be quite a considerable amount of traffic, but I have never tried running 300 odd players on an extDB so I am also not in the best position to comment on this as I believe the data access would directly depend on the database and modder. Even if this doesn't get published or pushed out it will still be interesting to know if I can get it right without any prior game development knowledge which is pretty much driving this atm.

We would be interested in helping you test it out. We currently have 8 servers to manage... it's a headache.

TOPArma Mission - Share your mission on GitHub