Using configuration files written in Python

Question

I noticed several Python packages that use configuration files written in Python. Besides the obvious escalation of privileges, what are the pros and cons of this approach?

Is there any advantage to this? Are there any guidelines on the best way to implement this?

Just to clarify: In my particular use case, this will only be used by programmers or people who know what they are doing. This is not a configuration file for software that will be distributed to end users.

Answer 1

The best example I can come up with for this is django settings.py , but I'm sure there are many other examples of using a Python file for configuration.

There are several key benefits of using Python as a configuration file over other solutions, for example:

• No need to parse the file:. Since the file is already Python, you do not need to write or import a parser to extract pairs of key values ​​from the file.
• Configuration settings can be more than just a key / values:. Although it would be foolish to have settings defining their own classes, you can use them to define tuples, lists, or settings dictionaries that allow more options and configuration than other options. This is especially true for django, where the settings file should correspond to all plugins that were not originally known to the developers of the framework.
• Writing configuration files is easy: This is false, but since the configuration is a Python file, it can be edited and debugged in the IDE of the program itself.
• Implicit error checking: If your program requires a parameter called FILE_NAME , and this is not in the settings, the program throws an exception. This means that the settings are becoming mandatory, and error handling in the settings can be more explicit. It may be a double-edged sword, but manual editing of configuration files should be for power editors, who should be able to handle the effects of exceptions.
• Configuration options are easily accessible and namespaces:. Having approached import settings , you can start calling settings.UI_COLOR or settings.TIMEOUT without any problems. They are understandable, and with the right IDE, tracking where these settings are made becomes easier than with flat files.

But the most powerful reason: Overrides, redefines, redefines . This is a rather difficult situation and may be case-specific, but one that is encouraged by django in several places.

Imagine that you are creating a web application with a development and production server. Each of them needs its own settings, but 90% of them are the same. In this case, you can do something like a configuration file that covers the whole development and make it (if it is safer) with the default settings, and then override it if it is created, for example:

 PORT = 8080 HOSTNAME = "dev.example.com" COLOR = "0000FF" if SITE_IS_LIVE: import * from production_settings.py 

Running import * from will cause any parameters that were declared in the production_settings.py file to override the declarations in the settings file.

I have not seen a best practice recommendation or a PEP document that describes how to do this, but if you want some general recommendations, django settings.py is a good example to follow.

• Use consistent variable names, preferably UPPER CASE, as they are understood as settings or constants.
• Expect odd data structures if you use Python as a configuration language, and then try to handle all the basic data types.
• Do not try to create an interface for changing settings, this is not a simple text editor.

When should you not use this approach? . When you are dealing with simple key / value pairs that should be changed by novice users. Python configurations are just a user parameter. Novice users will forget to end quotes or lists, and not be sequential, will delete options that they think are not applicable, will make the most irreconcilable errors and will only mix tabs and spaces ^only . Since you are mainly dealing with codes, not configuration files, all of this will break your program. On the other hand, writing a tool that will parse through a python file to find the appropriate parameters and update them is probably more of a problem than it costs, and you'd better use a reusable existing module like ConfigParser

Answer 2

I think Python code is directly used for configuration, mainly because it is just so simple, fast, powerful, and flexible. There is currently no other tool in the Python ecosystem that provides all of these benefits together. The ConfigParserShootout cat gives you enough reasons why it might be better to collapse Python code as config.

There are some security considerations that can be handled either by evaluating the security code, or by using policies such as properly configuring file system permissions during deployment.

I have seen so much struggle with a rather complex configuration, performed in various formats, using various parsers, but in the end, the easiest way to make the code.

The only real flaw that I came across is that the people managing the configuration should be aware of Python, at least the syntax, in order to be able to do anything and not slow anything down. It may or may not matter in each case.

Also, the fact that some serious projects like Django and Sphinx use this very approach should be quite reassuring:

• https://docs.djangoproject.com/en/dev/topics/settings/
• http://sphinx-doc.org/config.html

Answer 3

There are many options for writing configuration files with well-written parsers:

• ini
• json
• yaml
• xml
• csv

there is no reason for any configuration to be analyzed as a python script directly. This can lead to many problems: from security aspects to complex debugging errors that can be raised at the end of the program life cycle.

There's even discussion to build an alternative to setup.py for python packages, which is pretty close to a python-based source code configuration from the point of view of a python encoder.

Otherwise, you can just see python objects exported as strings that look a bit like json, albeit a little more flexible ... Which is fine if you don't eval() / exec() them or even import them, but pass it through a parser, such as 'ast.literal_eval' or parsing , so that you can only load static data, not executable code.

Only a few times when I realized that something close to the configuration file written in python is a module included in the library that defines the constants used by this library, intended for the user to process the library. I’m not even sure that it will be a good design decision, but I would understand that.

edit:

I would not consider django settings.py an example of good practice, although I believe that this is part of what I consider a configuration file for literate users, which works great, since django aims to be mainly used by coders and system administrators . In addition, django offers a configuration method through a web page.

To accept the @lego arguments:

• No need to parse the file

there is no need to explicitly disassemble it, although the costs of parsing are anecdotal, even more so taking into account security and additional security and the ability to detect problems at an early stage

• Configuration settings can be more than just key / values

ini files, you can define almost any basic python type using json / yaml or xml. And you do not want to define classes or initialize complex objects in the configuration file ...

• Writing configuration files is easy:

but using a good editor, the syntax of json / yaml or even xml can be checked and checked to have a perfectly processed file.

• Implicit error checking:

not an argument neither, as you say, is double-labeled, you may have something that understands well, but raises an exception after many hours of work.

• Configuration options are easily accessible and namespaces:

using json / yaml or xml, parameters can be easily placed in a namespace and naturally used as python objects.

• But the most powerful reason: redefines, redefines, redefines

This is not a good argument for python code. Given that your code consists of several modules that are interdendant and use a common configuration file, and each of them has its own configuration, it is quite easy to load the first main configuration file as an old old python dictionary, and other configuration files are simply loaded by dictionary updates.

If you want to track changes, there are many recipes for organizing the dicts hierarchy, which returns to another dict if it does not contain a value.

And finally, configuration values ​​changed at runtime cannot (actually should not) be serialized correctly in python, as this means changing the currently running program.

I am not saying that you should not use python to store configuration variables, I am just saying that no matter what syntax you choose, you must get it through the parser before getting it as instances in your program. Never download user modifiable content without double checking. Never trust your users!

If django people do this, it's because they built a framework that only makes sense when building a lot of plugins to create an application. And then, to configure the application, you use the database (which is a kind of configuration file ... on steroids) or the actual files.

NTN

Answer 4

I did this often in company internal tools and games. The main reason is simplicity: you simply import the file and do not need to worry about formats or parsers. Usually this was exactly what @zmo said, constants designed for non-programmers on the team to change (say, game-level grid size or screen resolution).

Sometimes it was useful to have logic in the configuration. For example, alternative functions that fill the initial configuration of the board in the game. I found this a big advantage actually.

I acknowledge that this can lead to difficult debugging problems. Perhaps in these cases these modules are more like game-level initialization modules than regular configuration files. In any case, I was very pleased with the direct way to create clear text configuration files with the ability to have logic there and did not receive a bit.

Answer 5

This is another configuration file option. Several fairly suitable configuration file formats are available.

Please take a moment to understand the point of view of the system administrator or the third-party vendor supporting your product. If there is another configuration file format, it may delete your product. If you have a product of monumental significance, then people will have difficulty learning the read-only syntax of your configuration file. (e.g. X.org or apache)

If you plan to use another programming language to access / write configuration file information, then a python based configuration file would be a bad idea.