For SkySpark we've been having a lot of various design discussions about the best to model and handle data quality issues. In the end we settled upon a design which boils all "bad data" down to a single special value which we call not-available or NA. This is essentially a special value like null that can be applied to any occurrence of other types such as Bool or Numbers in historical data. The NA value is a concept used by the programming language R.
As singleton value NA has some really nice advantages. First it simplifies applications which are visualizing or analyzing data - either you have a good data value (Bool, Number, Str) or you have a bad data value (NA). More importantly it allows compact representation when working with tabular time-series data as proposed by topic 347.
I would like to propose that NA be added as a new core data type. It is essentially a singleton value just like Marker.
Brian Frank Mon 11 Jan 2016
This is one component of Data Model 3.0 proposal.
For SkySpark we've been having a lot of various design discussions about the best to model and handle data quality issues. In the end we settled upon a design which boils all "bad data" down to a single special value which we call not-available or NA. This is essentially a special value like null that can be applied to any occurrence of other types such as Bool or Numbers in historical data. The NA value is a concept used by the programming language R.
As singleton value NA has some really nice advantages. First it simplifies applications which are visualizing or analyzing data - either you have a good data value (Bool, Number, Str) or you have a bad data value (NA). More importantly it allows compact representation when working with tabular time-series data as proposed by topic 347.
I would like to propose that NA be added as a new core data type. It is essentially a singleton value just like Marker.
Zinc encoding:
NAJson encoding:
"z:"