Hi all,

i just stumbeled across this behaviour in haxall. Is there any proper definition or documentation of the haystack filters "not" behaviour? (https://project-haystack.org/doc/docHaystack/Filters is not detailed enough)

pseudo code grid rows example: p = point, sensor, equipRef: someEquip, systemRef: [a,b] a = equip, system, air b = equip, system, water

Given a list of refs (here, systemRef) in a row, the following filter would apply as "any":

"point and systemRef->air"

For above example, this will return "p" as one of its system refs contains the air tag. So far, so good, and rather intuitive.

However, for the following query:

"point and not systemRef->air"

This still returns "p", while I expected the "not" to produce the negation of the part that comes after it, therefore returning nothing.

Is this a bug or is this intended behaviour, if so, is there any documentation on "not" behaviour when applied to trap paths?

On a side note, how can i distinguish ref and list of ref in the trio definitions, where one is allowed, and where not? https://project-haystack.org/doc/docHaystack/Filters#refLists refers to refs can be list of refs, but this is never to be used for "fundamental containment refs" e.g. siteRef, equipRef,...

systemRef are defined in trio as [^ref] def: ^systemRef is: [^ref]

while airRef as ^ref def: ^airRef is: ^ref

however, both are supposedly fine to use as list of refs aswell. The square brackets in "is" as i understood them signify something akin to multi inheritance, so that something is all of the listed. e.g. site: def: ^site is: [^entity, ^geoPlace]

so in either case, what does [^ref] in the systemRef definitions exactly mean?

thank you, cheers thomas

We can beef up the documentation. But the Haxall implemention is as your describe. You must have a complete path to the final tag or the filter evaluates to false. Then the only the leaf record is checked if it is missing the tag.

So given the filter: not systemRef->air, then any record missing the systemRef tag will evaluate to false. Only records that have a systemRef that points to a record missing the air tag will match.

The logic for the Filter behavior is here

On a side note, how can i distinguish ref and list of ref in the trio definitions

You cannot do this in the Trio defs today. But basically anything that uses the containedBy relationship cannot use a list, but anything that uses the inputs relationship can.

In the Xeto definitions the difference is formally defined by sys::Ref vs a sys::MultiChoice. Note the current tags compiled from defs have not been updated to use MultiChoice yet (will fix soon)

Hi! Thank you for the quick reply.

So given the filter: not systemRef->air, then any record missing the systemRef tag will evaluate to false.

This is perfectly clear.

My original question was: If my point has two refs in systemRef, and only one has an air tag:

The filter systemRef->air will return true for this point.

The filter not systemRef->air will also return true for this point in haxall.

Which is not what I expected, as my intuition tells me that the not applies to the boolean return value of whatever comes after it. This would also be my preferred behaviour, and loosely what https://project-haystack.org/doc/docHaystack/Filters suggests with "You can combine filters using and, or, or not" which reads to me that these operations apply to the filters, not inside the filters.

However, it could be argued that not systemRef->air is equivalent to systemRef->(not air) by returning true if any of the systemRef point to something that it not air. (Although I dislike this definition)

Anyways, playing around I now found that: I have a point with two refs in systemRef. Both systems have an air tag. not systemRef->air returns this point. It should not, in none of the above described applications of not.

I think this is a bug. Heres a test grid to reproduce:

{

"_kind": "grid",
"meta": {"ver": "3.0"},
"cols": [
],
"rows": [
    {"id": {"_kind": "ref", "val": "building_a"}, "site": {"_kind": "marker"}},
    {"id": {"_kind": "ref", "val": "ahu_a"}, "equip": {"_kind": "marker"}, "system": {"_kind": "marker"}, "ahu": {"_kind": "marker"}, "air": {"_kind": "marker"}}, 
    {"id": {"_kind": "ref", "val": "ahu_b"}, "equip": {"_kind": "marker"}, "system": {"_kind": "marker"}, "ahu": {"_kind": "marker"}, "air": {"_kind": "marker"}}, 
    {"id": {"_kind": "ref", "val": "some_sensor"}, "point": {"_kind": "marker"}, "sensor": {"_kind": "marker"}, "equipRef": {"_kind": "ref", "val": "ahu_a"}, "systemRef": [{"_kind": "ref", "val": "ahu_a"}, {"_kind": "ref", "val": "ahu_b"}]}, 
    {"id": {"_kind": "ref", "val": "some_other_sensor"}, "point": {"_kind": "marker"}, "sensor": {"_kind": "marker"}, "equipRef": {"_kind": "ref", "val": "ahu_a"}}  
]

}

with the query point and not systemRef->air I expect only to get some_other_sensor back, however, haxall reports both.

I can create an issue in the haxall github if that is desired, i just wanted to clarify the beaviour as defined in haystack first.

thank you! cheers thomas

Just a note the not operation really means missing tag rather than logical negation in Haystack Filters.

But you are definitely right, that was a bug in my Haxall implementation. I pushed a fix you can look at it.

I also updated the Xeto tag definitions to be MultiRef where appropriate in this changeset

hi!

wow, awesome! thank you. i checked out the main branch and that works now.

Just a note the not operation really means missing tag rather than logical negation in Haystack Filters.

ok, but that alone would not specify the behavior on multirefs.

For example: point and air and point and not air must result in disjoint sets. point and systemRef->air and point and not systemRef->air in my opinion should also result in disjoint sets. (but in haxall they dont).

For the behaviour that haxall currently implements i would expect a query in the form not systemRef or systemRef->(not air) (despite this not being a valid filter).

given this example grid: {

"_kind": "grid",
"meta": {"ver": "3.0"},
"cols": [],
"rows": [
    {"id": {"_kind": "ref", "val": "building_a"}, "site": {"_kind": "marker"}, "someBuldingValue": 234},
    {"id": {"_kind": "ref", "val": "a"}, "equip": {"_kind": "marker"}, "system": {"_kind": "marker"}, "air": {"_kind": "marker"}},  # noqa: E501
    {"id": {"_kind": "ref", "val": "b"}, "equip": {"_kind": "marker"}, "system": {"_kind": "marker"}, "water": {"_kind": "marker"}},  # noqa: E501
    {"id": {"_kind": "ref", "val": "some_temp"}, "point": {"_kind": "marker"}, "sensor": {"_kind": "marker"}, "equipRef": {"_kind": "ref", "val": "a"}, "systemRef": [{"_kind": "ref", "val": "a"}, {"_kind": "ref", "val": "b"}]}  # noqa: E501
]

}

I also dont see any use case for the current logic, as all use cases i currently have and i can think of require me to have not someRef->someMarker to be the logical opposite of someRef->someMarker.

E.g. Air refs: i want to find an equipment that does not directly connect to outside air. E.g. System refs: i want to find a temp sensor point that is not used in an ahu system.

These questions are currently not answerable in haxall.

If this is really the intended behaviour for haystack filters, this should documented that systemRef->air and not systemRef->air can produce overlapping sets as the not part does not apply the following statement as a whole but in case of trap operator(s) at any level of the trapped path and branches.

thank you, cheers, thomas

That isn't a valid hayson file so I can't really test with. Maybe post a zinc file.

But the way lists of refs are defined, its an OR operation against any of the refs in the list. So as long as one item in the list matches, then its overall match.

For example:

// given this record
systemRef: [@hasFoo, @missingFoo]

// this this filter....
not systemRef->foo

// ..essentially maps to this 
(not @hasFoo->foo) or (not @missingFoo->foo)

Although honestly using not should be strongly discouraged and we've been trying to remove its use in the ontology design. Its also very difficult to optimize because it requires a full database scan.

If this is really the intended behaviour for haystack filters, this should documented that systemRef->air and not systemRef->air can produce overlapping sets

This is correct