Choosing the right storage for configuration

Question

We have applications and services that use a large configuration, most of which are currently hardcoded in Java code and distributed across many classes. Obviously, this needs to be changed, and we want it to be centralized in one place and which must be retrieved and exposed by one of the services (e.g. ConfigurationService), which also includes configuration caching for its clients to improve performance. We also need to have a dynamic reload of the configuration for long-term applications and avoid a restart. I would like to get a few comments about which storage I should use for this purpose -

Data Type -

Data should not be structured. It can be a simple key-value pair, or it can be a multi-tasking pair with a single value. Here are some random configuration examples -

• "Client_Id" → "ABC" (Simple Key-Value)
• ["User_Type": "Admin", "Region": "Mumbai", "User_rating": "9"] → ["Commission": "10%"] (value with several keys)
• ["User_Id" → "123"] → ["WhitelistedRegions" → "Mumbai, Goa"] (Context based configuration, value is a list)

Storage Solutions -

In principle, the set and type of parameters that form the key are not fixed, which gives me a hint that this configuration is not really structured. The volume of the entire configuration will not really be huge. Compared to readings, there will be very few entries.

• Database (RDBMS / NoSQL) . The advantage of using a database table can be security and the backups that it provides. Since this is not like relational data, I would consider a NoSQL solution. I have not personally used them personally, please tell me which one is best for this kind of data. Since there can be many different keys, we should be able to select the exact keys (some indexing). Using the database will lead to a delay, but effective caching can be created to overcome it (since there will not be too many entries in the configuration). Data is easier to request.

• Files (XML or other flat files) . We can store it simply using files. Caching can also be used with files. While the entire configuration can be imported into memory (RAM), this option should also be implemented (exception of selective cache). Files provide version control, permissions / security. XML files can get messier if they get large. Data may not be easy to request if we use files.

What should be the best storage solution, assuming that dynamic reloading and cache invalidation are implemented separately? What other factors should be considered here?

If files for storing such a configuration are used, then which files are better suited for such use cases?

Note. I asked a similar question on SO, but I probably did not formulate the question as clearly as I should, so I created a new one, instead of making heavy changes.

Answer 1

Please do yourself a favor and appreciate whether Archaius or Curator is right for your needs. Archaius is probably more suitable for configuring applications and containers, where the Curator is probably better for adjusting the machine level.

Answer 2

The examples you presented suggest that you might need some kind of rule engine. To show what I mean, I interpret your examples as having the following semantics:

if (true) { Client_Id = "ABC"; } if (User_Type == "Admin" && Region == "Mumbai" && "User_rating" == "9") { Commission = "10%"; } if (User_Id == "123") { WhitelistedRegions" = ["Mumbai", "Goa"]; } 

If my interpretation is incorrect, perhaps you can edit your question to clarify your intended meaning. On the other hand, if my interpretation is correct, then I don’t know any specific configuration syntax specifically designed for your requirements. Instead, I suspect that you will have to tailor the semantics of what you want into the limitations of any configuration syntax that you decide to use.

How can I try to explain (your interpretations) your examples in the Config4 * syntax (disclaimer: I am its main developer) is as follows:

 uid-rule { # unconditional client_Id = "ABC"; } uid-rule { condition { User_Type = "Admin"; Region = "Mumbai"; User_rating = "9"; } Commission = "10%%"; } uid-rule { condition { User_Id = "123"; } WhitelistedRegions = ["Mumbai", "Goa"]; } 

I recommend that you read Chapter 2 of the Config4 * Getting Started ( HTML , PDF ) manual so that you can understand the syntax used in the above example.

My initial attempt to train your examples in XML syntax:

 <rules> <rule> <property name="client_Id" value="ABC"/> </rule> <rule> <condition name="User_Type" value="Admin"/> <condition name="Region" value="Mumbai"/> <condition name="User_rating" value="9"/> <property name="Commission" value="10%"/> </rule> <rule> <condition name="User_Id" value="123"/> <property name="WhitelistedRegions" value="Mumbai, Goa"/> </rule> </rules> 

Please note that neither the Config4 * parser nor the XML parser will provide you with the semantics that you want to get out of the box. Instead, you should write a class called RulesEngine . Such a class: (1) analyzes the configuration file to obtain rules and caches the analyzed representation in memory; (2) provide a simple API for requesting this set of rules in memory; and (3) provide a reloadConfiguration() method that reassembles the configuration file. Your application will somehow call the reloadConfiguration() method call (for example, every few minutes).

If you use XML for configuration syntax, I suggest that you achieve the goal of centralization by storing the XML file on a web server. An XML parser can get the file form there. If you use the Config4 * syntax, then integrating Config4 * with curl allows you to do the same.

Answer 3

There are different ways to look at a situation.

• How can I decide where to store the data, usually based on the data type.

  If I read your data correctly, it indicates the customer ID, the commission based on the user's location and rating, and the regions applicable to a particular user. For me, this is more of a static business data than a configuration (although this will not change frequently). Therefore, I would prefer to store it in the database.

• I need to provide a screen for users to update this data. If there is a user interface, you do not need to go through the process of raising a ticket or issuing it in order to do this in PROD env. But having a user interface will move the selection to the database.

• If there is no user interface, it might be better to choose a file configuration. Because you can have a file viewer service in your application to view this configuration file. After changing the file, this service will read the file and update the configuration without rebooting. Dynamic rebooting in the case of a database may not be so simple. You can still provide an administrative screen for manually starting a configuration reload.

• For the file type, you can choose the option suggested by Ciaran, but this will require additional training. Given that you don't have a lot of configuration, you can stick with xml.

• Caching data, which I think will not have any effect on the choice between database and file, as this is easily possible for both. However, if you use some kind of ORM, such as hibernation, you will be protected from the extra effort of caching your data, but ORM brings another difficulty in defining class mappings to database tables.