Good morning everyone.

The pyhaystack project is living again ! Thanks to @sjlongland and @sudo-Whateverman (see profiles on Github) for their very appreciated, technical contribution to a project that has been sleeping for too long.

Come see us on Github if you want to participate.

Project is re-starting but I'm confident it will be a really big addition to project-haystack.

Python is more and more appreciated and recognized as a tool of choice in the data science community and with all the great python tools available (think pandas, numpy, matplotlib or bokeh), it suits very well to the goal of this project which is extracting facts and knowledge out of your building data... and being able to see all that in Jupyter Notebook is quite cool.

See you there

Just a heads up to those who might be watching here.

We're in the process of doing a ground-up re-design of the API. Two things would greatly help here:

• some ideas on peoples' actual use cases for Haystack clients, with a view to try and tailor pyhaystack to meet those needs.
• any gotchas regarding specific implementations of Project Haystack that we need to be wary of.

An example of the latter is in the use of JSON. Some implementations don't support it, some support it but use a completely different dialect of it, and some use the standard dialect as documented on the Project Haystack website.

ZINC seems to be a lowest common denominator, which is why I wound up creating hszinc (also on pypi) to take care of parsing grids in the ZINC format as well as handling the standard JSON grid dialect.

It seems to be coping with most use cases, but as a clean-room implementation, it could use some real-world testing. I already found a number examples of the ZINC format that are invalid according to the grammar mentioned on the ZINC page, and so if people find issues, that would help the project along too.

As for use cases, pyhaystack has previously been used as a means for acquiring data for numerical analysis, and I believe this should continue. I'm also looking to other use cases such as background data collection or uplink services.

In short though, if there's a particular need you're trying to solve with Python and Project Haystack, now would be a good time to pitch in and help us. :-)

Regards, Stuart Longland

some ideas on peoples' actual use cases for Haystack clients

I would say the most common case is:

1. learn about the points available in server
2. read those point history data
3. subscribe to those points via watches

   I already found a number examples of the ZINC format that are invalid according to the grammar mentioned on the ZINC page

I think it would be great if when you encounter inconsistencies in either implementations or documentation that we capture it so that we can fix one or the other

latter is in the use of JSON. Some implementations don't support it, some support it but use a completely different dialect of it

I think this is just timing since it was pretty recently that we finalized a JSON format that provides 100% fidelity. Going forward JSON should be as well supported as Zinc (easier to tokenize by using a library, just more verbose)

I was curious to the status of this project.

Use case here is for Volttron a DOE BAS type project for AFDD, DR, Historian and Analysis features. Built on python and I would like to extend a natural way for the system to communicate to a haystack server. Particularly would like to see a client build for serving info from an embedded linux device as a Haystack device for data collection.

https://github.com/VOLTTRON/volttron

More info on the project.

@Justin the project is about to be very active.

There have been multiple discussions about the direction of the project and we are about to begin the implementation.

You can go here : https://github.com/ChristianTremblay/pyhaystack

And there's a gitter chat room where you can ask specific question and get involved in the discussion.

https://gitter.im/ChristianTremblay/pyhaystack

Personally, I'm actually using the branch called Mixin_issue6 https://github.com/ChristianTremblay/pyhaystack/tree/Mixins_issue6?files=1

It implements 3 clients (I use AX, but there's a widesky and skyspark client there)

I use it in a synchronous way using Jupyter...

But as decided in our plan, we'll begin to implement asynchronous features (while keeping the sync feature alive for "on the go" analysis).

Best would be to come and join us on gitter.

Hoping to see you there.

Thanks for the info! I will follow and see what I can bring to the table. Pass this along to the DOE/PNNLab guys to see if they can lend some expertise on this. Trying to get them to bite on Haystack being a defacto tagging method on their data.

For what it's worth Justin, that's our immediate use case for pyhaystack too.

Basically we've got embedded Linux data collection devices and will be pushing data via a Project Haystack API.