Geek Answers Handbook

Is it possible to have an optional field in an Avro schema (i.e. the field does not appear at all in the .json file)?

Is it possible to have an optional field in the Avro schema (i.e. the field does not appear at all in the .JSON file)?

In my Avro scheme, I have two fields:

{"name": "author", "type": ["null", "string"], "default": null}, {"name": "importance", "type": ["null", "string"], "default": null},

And in my JSON files, these two fields can exist or not.

However, when they do not exist, I get an error message (for example, when I test such a JSON file using the avro-tools command-line client):

 Expected field name not found: author

I understand that as long as the name field name exists in JSON, it can be null or a string value, but what I'm trying to express is that JSON is valid if the names of these fields do not exist , OR if they exist, and they are null or string. "

Can this be expressed in the Avro scheme? If so, how?

+9
json avro
source share
2 answers

According to the avro specification, this is possible using the default attribute.

See https://avro.apache.org/docs/1.8.2/spec.html

default: the default value for this field, used when reading instances in which this field is absent (optional). Allowed values ​​depend on the type of field layout according to the table below. The default values ​​for the join fields correspond to the first schema in the join.

In your example, you add a default attribute with a value of "null", so this should work. However, support for this also depends on the library that you use to read the avro message (there are libraries in c, c ++, python, java, c #, ruby, etc.). Perhaps (possibly) this function is missing in the library you are using.

+2
source share

You can define the default attribute as an undefined example. therefore, the field can be skipped.

{ "name": "first_name", "type": "string", "default": "undefined" },

Also, all fields are required in avro. if you want it to be optional, combine its type with zero. Example:

  { "name": "username", "type": [ "null", "string" ], "default": null },
0
source share

More articles:


All Articles