Welcome, Guest. Please login or register.

Pages: [1]
Print
Author Topic: Objects DB  (Read 7102 times)
b0rland
Community member

Posts: 105



View Profile Email
« on: January 10, 2010, 05:35:15 PM »

To keep the forums alive, I'm posting some routine notes and considerations about objects database in case someone wants to comment or argue.

Brief introduction into the topic.
The need to store certain attributes for various object types arose gradually. For now some of these data is scattered around the project tree: some of it is in object definition xmls, some is in map files, some more is hardcoded, and much more is about to come in future. So the generally accepted idea was to make a separate objects db to store those attributes. This way we can store all the attributes common to a certain object type in a single place -- the objects db. These values can afterwards be overridden by values specified in a map file. I'm hoping this would also allow to eliminate the object definition files in their current form (or rather merge with them), but I didn't study that part closely yet.

General design considerations
By flipping a coin (not really, but I admit the certain randomness of the choice) I chose to use YAML instead of XML. Thanks for Maximinus for the idea. I also intend to follow the 'one object type per file' convention. It's good enough for our current amount of object types. In future we'll probably want to refactor it and learn to group multiple object types in the same file, while still supporting multiple YAML files, but we might as well arrive at a completely different storage format by then, so I don't intend to hurry things up.
Programmatic design of this db is still not clear. I'm going to post updates into this thread after I have it drafted.

Planned YAML structure and example
The example below is self-explanatory:
Code:
## Example YAML
---
OBJECT: dagger
PYTHON_CLASS: MeleeWeapon
PROPERTIES:
  name: dagger
  description: A handmade dagger. Ugly thing, but still deadly in skilled hands.
  weight: 0.3
  bulk: very_small    #corresponding numerical value = 3^1
  inventory_image: "gui/inv_images/inv_litem.png"

The map-related properties from current object description xmls may form additional MAP: section once they are merged. In the nearest future we will probably want to introduce inheritance so that object types could inherit all the properties from certain base type and override some of them. This can be done by creating INHERITS: <base_object_type> section.

Some things I specifically decided not to do. I decided not to split properties into smaller categories: basic/extended or based on their respective python mixin (carriable/container/openable/etc). Although it may look like a tempting idea, I don't see any specific advantages that it would bring. Again, in future it may be reconsidered. Probably something else, but couldn't remember Smiley.

As usual, any feedback is highly appreciated.
Logged
shevegen
Community member

Posts: 705



View Profile
« Reply #1 on: January 11, 2010, 12:41:07 AM »

Seems pretty easy. I have a few questions, they may sound silly, but still here goes:

- If we would use many many many yaml files, would this negatively affect performance?

- Are the quotes needed?
  inventory_image: "gui/inv_images/inv_litem.png"

I think this here should work as well or?
  inventory_image: gui/inv_images/inv_litem.png

- Come to think about it, perhaps the path does not have to be absolute? If all objects reside in a specific dir, we dont have to use "gui/inv_images" part or? We could stick to the file name.

I didn't understand the INHERITS example Smiley
Could you give a specific example of what you mean with inherit? Inherit as in:

object_1:
stone
object_2:
knife INHERITS from stone

(a stone knife)

?

Quote
I chose to use YAML instead of XML.

Actually, we could generate the XML via a python script too and stick to use YAML (which is less annoying because you dont have to use all those <> tags)
Personally I no longer use XML - I found it way too verbose in the end, especially compared to YAML and JSON.

Quote
'one object type per file' convention.

Sounds good. Easy standard, helps avoid confusion or make things too complicated.

Quote
group multiple object types in the same file

Hmm, I guess for now we can stick to something simple. What I think... in the future... might be worth to look at, could be guns and rifles. Let's say we would have 30 different guns ... would we put them into a new directory? Or all in one yaml file? I am not sure here. I think the approach to use one yaml file / object might be easier and better, but it could get out of hand when we reach like... 1000 objects or something.
Logged

Cleaning away the bureaucracy in PARPG to make our life easier.
maximinus
Community member

Posts: 694



View Profile Email
« Reply #2 on: January 11, 2010, 02:21:41 PM »

My only concern right now - and I admit, it's a big one - is in how we are currently using YAML in the dialogue and quest files. Ideally, we'd use the same text format everywhere, but FIFE itself uses XML, and it seems that many of us would not choose this format for various reasons. Luckily the use of XML is pretty limited.

However, as the quest and dialog scripts get bigger, we are going to meet a problem, in that YAML is not suited to storing chunks of code, and I think we will end up using ANOTHER data format somewhere along the line (or implementing some mini-language in YAML).

Also, we'll need most objects to be tied to quests (i.e. contained in the quest files data) so that it's easy to remove / swap out quests in an instant.

So, putting these two together - and assuming that XML is out - I reckon the choices are:

  • Python - +Trivial to use -Hard to sandbox
  • YAML +Already in use -Ugly at storing code
  • Other +Could solve all issues -It's new

My suggestion is to use S-expressions, as suggested here - http://forums.parpg.net/index.php?topic=619.0. It's trivial to write a parser for this, and both data and code can be easily written this way.

Now I'll admit that I wasn't around the for the initial YAML discussion a few months ago but from reading previous posts it was picked because it was easy to parse and looks good (can't argue there). I just don't think it'll be good for some of the code-driven quests that we will end up with in the future.

By flipping a coin (not really, but I admit the certain randomness of the choice) I chose to use YAML instead of XML. Thanks for Maximinus for the idea.

The idea for Yaml, or for flipping a coin?   Tongue
Logged

Science is open-source religion
shevegen
Community member

Posts: 705



View Profile
« Reply #3 on: January 11, 2010, 09:38:16 PM »

Quote
Now I'll admit that I wasn't around the for the initial YAML discussion a few months ago but from reading previous posts it was picked because it was easy to parse and looks good (can't argue there).

From my perspective I think two programmers had different ideas here but at the end of the day, the implementation that seemed to work better at that time was used. Perhaps I misremember, but I really think that the yaml dialogue parser thingy was more advanced at the time.

Like, you know, a do-acry - who gets to push forward with something, will see it implemented first.

My basic argument back then actually was that "we could have more than one implementation and see what works".
I am fine with any approach here, they don't necessarily have to be mutually exclusive. It's just that, if I have the choice between XML and YAML, I would always pick YAML. That being said, I am not 100% happy with the yaml dialogue files. It seems to be partially mixed with python code already as well.

Note - this is not a strong dislike on my part at all. I just felt that it could be better, somehow, when people create plots and such. Also, what we should really keep in mind is providing a GUI. If we provide a GUI, then the underlying mechanism doesn't matter that much. People would not have to care if they edit a S-expression file, YAML file or XML file.

I have read through http://forums.parpg.net/index.php?topic=619.0

Ultimately, I think we should aim for a lot of flexibility in quests. You gave the example of the Megaton quest, where more or less enemies are spawned depending on the choice of the actor (or his position). You also gave the example of updating dialogue files "on-the-fly" (or, depending on things that happened)

My head is a little bit dizzy because keeping too many things in my head makes me confused Cheesy

If the S-expression allow an easier way to handle all that logic parts (logic parts of handling dialogues, quests, npc interacting and such) then it would be a better choice than to use YAML or?

What would be kinda nice would be to be able to compare such a parser to the yaml dialogue format side-by-side. Right now, for example, I have no real idea how that S-expression would look or if it is easy to express something in it
Logged

Cleaning away the bureaucracy in PARPG to make our life easier.
b0rland
Community member

Posts: 105



View Profile Email
« Reply #4 on: January 11, 2010, 11:30:32 PM »

Maximinus, I certainly agree with what you're saying except for this little point:
Also, we'll need most objects to be tied to quests (i.e. contained in the quest files data) so that it's easy to remove / swap out quests in an instant.
At an early stage it will probably be true. But in a fully grown game hardly 1/10 of all objects are tied to quests. Some are just weapons. Some are just useless junk introduced to add some colour to the game. I don't think it makes any sense to store those together with quests. I rather see it as a separate db of some sort. And it should be simple in structure and optimized for maintainability (because the number of objects will start growing exponentially at some point). In that respect I don't think s-expressions are the best way of representing a database of any kind. I'd vote for YAML or even some binary format of flat DBs storage like Tokio Cabinet.
On the other hand, quests will probably heavily reference the objects and maybe conditionally endow them with certain unique properties uncommon to the corresponding class of objects in general. I.e. extend the set of attributes initially fetched from DB for some specific object instance. This functionality doesn't contradict anything and it can be accessed from s-expressions directly if we choose to use them as the storage engine for quests.
By flipping a coin (not really, but I admit the certain randomness of the choice) I chose to use YAML instead of XML. Thanks for Maximinus for the idea.
The idea for Yaml, or for flipping a coin?   Tongue
The idea of YAML of course. I choose to reserve a copyright for coin-flipping Smiley
Logged
zenbitz
Community member

Posts: 1164



View Profile
« Reply #5 on: January 12, 2010, 12:12:01 AM »

yes, I agree "most" objects will be very standard with some particular features and no "quest" interactions.

YAML is only slightly more human writable than XML (did you know you have to have the indents aligned).  I am not sure what the advantage of the S-constructs are over XML... they seem slightly less verbose but with () instead of <></>. 

There are numerous freely available XML readers/editors.

At this point - I cannot imagine a non-programmer HAND writing the scripting bits.  So they may as well be python.  For any real non-programmer there will have to be a GUI - in which case the file format becomes nearly irrelevant.

I have no problem with XML as a data definition system; I agree it sucks somewhat for code fragments.

It makes no sense to me to NOT write the scripts in python since python is a very easy language and if it has bonus features - who cares, just don't use them.    I don't see how "hard to sandbox" is a problem, just write a small sandbox app that handles the quest/dialog stuff (we actually already have this in part.  But maybe I don't understand what you mean by "hard to sandbox".

In the end - all the "content" files are going to have to be compiled into something binary so that the loading times aren't astronomical (at least this is the info I get from people in the industry)

Logged

We are not denying them an ending...
We are denying them a DISNEY ending - Icelus
maximinus
Community member

Posts: 694



View Profile Email
« Reply #6 on: January 12, 2010, 03:16:00 AM »

Maximinus, I certainly agree with what you're saying except for this little point:
Also, we'll need most objects to be tied to quests (i.e. contained in the quest files data) so that it's easy to remove / swap out quests in an instant.
At an early stage it will probably be true. But in a fully grown game hardly 1/10 of all objects are tied to quests. Some are just weapons. Some are just useless junk introduced to add some colour to the game.

Ok, perhaps I should re-write that, what I meant was quest scripts will need to have data items in them (to swap them in and out easily), so I'd prefer to use the same format for both things (quest data and normal items data).

It makes no sense to me to NOT write the scripts in python since python is a very easy language and if it has bonus features - who cares, just don't use them.    I don't see how "hard to sandbox" is a problem, just write a small sandbox app that handles the quest/dialog stuff (we actually already have this in part.  But maybe I don't understand what you mean by "hard to sandbox".

In the end - all the "content" files are going to have to be compiled into something binary so that the loading times aren't astronomical (at least this is the info I get from people in the industry)

Many (!) months ago I had a similar discussion with tZee and tie of a similar nature, where I threw out the idea of building objects on the fly by using eval() or even the python new module, but they saw this (the mixing of data + python) as a serious security issue. I didn't because the data and the script both come from the same source (us), but memory of that discussion made me bring up this point. Is it a concern of the current programmers?

If not, then - the more I think about it - it makes sense to me that we just stick with Python. We all know it, we don't need any extra libs, no need to parse anything and it can handle data and code just fine. The 2 issues i think of are:

1: You could write bad code that trashes the quest data, or loops forever
2: Slightly harder to machine write (i.e, get a nice GUI to output)

My other proposed option, using S-expressions, solves the above but has it's own disadvantages:

1: It's different from Python
2: More code (the parser) -> more bugs

Obviously, we would need to write a little package to make quest scripting easier (a lot of helper functions) but that's not hard work really.
Logged

Science is open-source religion
maximinus
Community member

Posts: 694



View Profile Email
« Reply #7 on: January 12, 2010, 07:22:14 AM »

Here's a few more of my thoughts. As to what should / shouldn't be in the techdemo, see the new thread I put up.

All the non-fife data should be in gamestate. That is:

The PC
The NPC's
The objects
The quests
The dialog

We should be able to pickle and unpickle the gamestate to save and load the game.

Every item should have a unique, text based ID, and we use this ALL the time to reference the object / item / npc.

Rationale: +We pickle this, not the reference
                +Helps debugging
                -Slower, more memory
 
The base map file should contain only those things that are needed by fife. Another file should contain the data needed by the gamestate class.

Rationale: +We avoid using XML
                +Objects can have custom scripts
                -We'll have to sync them using a script

Finally, here's what needs to transferred to whatever script we use in the end, all the data that's spread about the code and maps:

Line 124 of /scripts/objects/actors.py defines items on the character at the start
2 items - the containers - are built into /scripts/objects/containers.py
There's a door defined in /scripts/objects/doors.py

In the map file:
On line 3452 and on, data for the NPC's on the map.
On line 5472, data for the container

There is a file /scripts/items.py that is not used by anything. Anybody know why?
Logged

Science is open-source religion
b0rland
Community member

Posts: 105



View Profile Email
« Reply #8 on: January 12, 2010, 11:09:03 AM »

Ok, perhaps I should re-write that, what I meant was quest scripts will need to have data items in them (to swap them in and out easily), so I'd prefer to use the same format for both things (quest data and normal items data).
Not quite sure what you mean here. An example could prove useful.

Many (!) months ago I had a similar discussion with tZee and tie of a similar nature, where I threw out the idea of building objects on the fly by using eval() or even the python new module, but they saw this (the mixing of data + python) as a serious security issue. I didn't because the data and the script both come from the same source (us), but memory of that discussion made me bring up this point. Is it a concern of the current programmers?
I don't see it as a security hole at least. And I think it can be made to look neat enough too. But again I suggest we don't start major changes in quest engine on the eve of techdemo.

Every item should have a unique, text based ID, and we use this ALL the time to reference the object / item / npc.
Rationale: +We pickle this, not the reference
                +Helps debugging
                -Slower, more memory
Well, why can't we have the best of both worlds? We already have unique ids for each physical in-game object. I don't think PARPG has a reference table (id -> object instance), I suppose FIFE does. If not, this thing is essential anyways, so we'll have to come up with it sooner or later. But my main point is that we don't need to reference things by IDs (=> table lookup each time we reference an object => slow and confusing code), only to be able to pickle them this way. If memory serves, pickle lib allowed to customize the way objects are pickled. So we can easily convert references to IDs when pickling and backwards when un-pickling.

Finally, here's what needs to transferred to whatever script we use in the end, all the data that's spread about the code and maps:
The rule of thumb I'm using is that object classes -- which vary greatly in their behaviour, like weapons and containers, firearms and knives -- should be defined as python classes, with some hierarchy. Object types -- which are more specific then classes, but only vary in their parameter values, like "colt .44" and "firelock" (both of the class 'firearms') -- should be defined in objects DB, whichever format it ends up using to store them. That's where class name and parameters for constructor will be taken from.
And there should be a way to customize every individual instance of object type so that things like "Snanty door" didn't have to exist as a separate class or even separate objects DB records. These things can be done on a map -- which is currently the only place where all the object entities with their IDs are being stored.  I don't really see a reason for splitting the map into two files. As long as common objects definitions are all in object DB, the map only contains FIFE data (coordinates, etc) and whichever custom values we want to assign to a specific instance, if any.

Based on the above:
2 items - the containers - are built into /scripts/objects/containers.py
There's a door defined in /scripts/objects/doors.py
ImmovableContainer class is enough for containers and Door class for doors. Specifics for each type of container/door goes to objects DB. In the map file we are to specify the contents of each crate, the from/to values of the door, coordinates and IDs of all three.

In the map file:
On line 3452 and on, data for the NPC's on the map.
On line 5472, data for the container
I don't see anything bad about those two. Except for unused attributes that keep being repeated, but I guess that's a flaw in map editor, not the map format itself. I think having all the object instances in a map file (and editable through map editor) is a good idea in general.

There is a file /scripts/items.py that is not used by anything. Anybody know why?
This one is sort of a pre-historic objects DB from the time when all objects in the inventory were just images. There was little use in it after we switched to actual objects, but I decided to keep the file for possible future reference and maybe to re-use the same file for new objects DB.

« Last Edit: January 12, 2010, 11:13:18 AM by b0rland » Logged
maximinus
Community member

Posts: 694



View Profile Email
« Reply #9 on: January 12, 2010, 01:18:16 PM »

The mixing of data + python as a security issue; Is it a concern of the current programmers?
I don't see it as a security hole at least. And I think it can be made to look neat enough too. But again I suggest we don't start major changes in quest engine on the eve of techdemo.

Nor do I. Phew!  Smiley

Every item should have a unique, text based ID, and we use this ALL the time to reference the object / item / npc.
If memory serves, pickle lib allowed to customize the way objects are pickled. So we can easily convert references to IDs when pickling and backwards when un-pickling.

Ok, sounds like a better idea, and is still simple.

I think having all the object instances in a map file (and editable through map editor) is a good idea in general.

Ok, and now here's for that example you asked for. Obviously you are in favor of having all the data for any given map held in the mapfile. That's logical, it means we only have one map file to deal with, and we already have an XML parser.

The part I'm thinking of is this: when we have a new quest, we are going to want to separate out any extra items enclosed in the quest so that it becomes easy to 'pull out' the items, if we remove the quest, or to edit them easily. For example, the current map file (and one that I don't think is that complicated right now) is a massive 5787 lines long, not something I'd like to work with. So either we separate out special quest items, or we build some scripts to help us handle the complexity in some way. This second file has to have some format or other, I was suggesting that it have the same format as the quest engine scripts (so you can mix the 2 - or combine them! - easily). This format, whatever it was, would NOT be xml (since code definitions in XML look ugly).

I hope that makes sense.
Logged

Science is open-source religion
Pages: [1]
Print
Jump to: