Having an easy way how to validate messages against given version makes sense to me. For keeping old versions of spec in git I propose to use tags, e.g. 0.1.2 to make the maintenance easy and efficient. I see two ways how to "unpack" all supported versions:

• During packaging checkout every x.y.z tag and make clean; make convert
• Dynamically unpack requested version upon validation call

The second option could work like this:

validator = ci.Validator(url='https://pagure.io/fedora-ci/messages/')
validator.validate(topic='...', message='...')

Validator would git clone the latest specification from the provided url, check version from the message, git checkout version, convert schemas and cache them for future use so that subsequent calls are fast. What do you think?

i am fine eith option 2 if it will be cached and cloned just once ... maybe @happz has some ideas also how would it be most reasonable ...

I'd opt for option number 1 - making the process depending on being online at least once for each version seems to me as complicating things for lesser gain. After all, the schema files are just plain text, compressing them shouldn't be hard, but I don't think they would occupy too much space anyway. And the whole installation and runtime could be offline, ready to go out of the box.

My goal is to use this in multiple services, maybe even with higher traffic, tests and so on. The simpler and fool-proof installation and use, the better. I believe fetching schemas when necessary adds yet other needles moving bit that could break.

I see. Make sense.

So, fedmsg is being replaced by fedora-messaging, and fedora-messaging is very much in favour of message schemas. In fact, in fedora-messaging, if a message declares a schema and a Python version of that schema is installed on the broker, the message will be validated against the schema definition before it's published; if a message declares a schema and a Python version of that schema is installed on a system where a consumer runs, the consumer will get the message as an instance of the schema. The schema can even provide methods for the consumer to use.

Here's the fedora-messaging docs on this. You can find real-world schema examples in Bodhi . Bodhi is publishing fedora-messaging messages in production, and using the schemas defined there; if you write a consumer of Bodhi messages and run it somewhere with python3-bodhi installed, the consumer gets the messages as instances of the relevant schema class. For example, bodhi.compose.complete messages appear as instances of bodhi.messages.schemas.compose.ComposeCompleteV1 and you can e.g. get a summary of the message as the message.summary attribute (implemented as a property in the schema class). Another example: I've got a pending PR to implement schemas for the bugzilla2fedmsg project that emits messages (previously fedmsgs, now fedora-messaging messages) in response to Bugzilla events.

The fedora-messaging schemas are based on JSON Schema, so there's obviously compatibility/overlap here, and a goal of this project is that Fedora test systems emit messages in the formats defined in this spec. Given this, it'd be great if this project could be extended to provide fedora-messaging schema implementations of the schemas, such that they can be used for publishing and consuming fedora-messaging messages...

@jcline

I don't have a ton to add other than I think the fedora-messaging APIs should cover your needs. You can manage versions of your schema by versioning the Python package with the schema in it and it gets you the simple "pip install" you want.

So I'm not sure if this exactly matches @happz 's use case, but until he tells me otherwise I'll keep posting here :)

After a bit of a chat with @jcline and @ralph I wrote a PoC for a Python module that's essentially a fedora-messaging schema wrapper for ci-messages. Here it is. The README explains how it works. If we like this approach, I can send a PR for this project's Makefile to install the converted schemas to the path my module defaults to looking in (/usr/share/ci/messages). For trying the module out for now, you can just run make convert on a local checkout and use a config file that points to the location of the resulting JSON files.

For now this doesn't really address versioning. There are various ways we could do that, but I wanted to just do this much for now.

oh, note, for now to handle the references in the schemas it uses the jsonref module. That's not yet packaged for Fedora but is available from pip. We could avoid this by defining a custom validate method in the schema class that uses a RefResolver like validate.py does, but I didn't really want to do that for now, I wanted to prove this can work with the stock fedora-messaging validate().