Welcome to inicheck’s documentation!¶

Contents:

Welcome to inicheck¶

https://img.shields.io/pypi/v/inicheck.svg

What is inicheck?¶

inicheck is an advanced configuration file checker/manager enabling developers high end control over their users configuration files. inicheck is specifcially aimed at files that adhere to the .ini format. See wiki for more info on the file standards https://en.wikipedia.org/wiki/INI_file .

Objectives¶

The goal of inicheck was to reduce endless logic gates and repeat code for software that required complicated configuration files. We also wanted to centralize options for an ever changing repo. We have a lot of experience with codes that required config files that could easily have 1000 different configurations. So from that need inicheck has been born.

Basic Principles¶

inicheck requires a master config file to operate. The master config file is a gold standard files provided by the developer of a package. The master config (sometimes called the core config) provides the rules for every configuration file that is passed to the software. There the developer can set types, constrain options, documentation strings and defaults. The developer can also create recipes that can be triggered when a user has a specific combination of config file entries.

Once a master config file is created, all config files can be passed through the standards laid out in there. From there the software can provide feedback to the user about why their file is being rejected. The master file also enables the user to have really simple config files that have larger meaning. With recipes and defaults, a user can provide relatively little and the developer can still move forward without have a ridiculous amount of logic to handle scenarios.

Free software: GNU General Public License v3
Documentation: https://inicheck.readthedocs.io.

Features¶

.ini file generating
Config file checking
Config file warning and error reporting
Command line interface
Apply defaults
Auto cast items into the correct types.
Constrain users by designated options
Create recipes to ensure the right settings are in place
Use custom types for your config file
Use multiple master config files for cleaner files
Auto config file documentation
Create changelogs to automatically manage deprecated config options

Credits¶

inicheck has been developed at the USDA-ARS-NWRC during efforts towards creating highly configurable, physically based watershed modeling software.

inicheck originally was based on config parser and with time went away from it. We would not be here with that base and for that were grateful.

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Installation¶

Stable release¶

To install inicheck, run this command in your terminal:

$ pip install inicheck

This is the preferred method to install inicheck, as it will always install the most recent stable release.

If you don’t have pip installed, this Python installation guide can guide you through the process.

From source¶

The sources for inicheck can be downloaded from the Github repo.

You can either clone the public repository:

$ git clone git://github.com/USDA-ARS-NWRC/inicheck

Or download the tarball:

$ curl  -OL https://github.com/USDA-ARS-NWRC/inicheck/tarball/master

Once you have a copy of the source, you can install it with:

$ python setup.py install

Usage¶

To use inicheck in a project you will need to create a master config file for all your configuration files to be checked against. To get started with writing a master configuration file see Master Configuration File.

Command Line¶

A First Use Example:¶

Once the Master file is created, create a config file and check it! Consider the following example. Lets say the Master Configuration file contains one section and one recipe like:

[settings]

  username:
          default = guest

  age:
        type = int,
        description = users age

  job_class:
            default = grunt,
            options = [grunt middleman exec],
            description = class of the users job to limit access

[settings_recipe]

    trigger:
	    has_section=settings

    settings:
    	apply_defaults = True

To make a simple configuration file we need to create the file and fill out whatever entries are important to the specific user:

[settings]
  username: coolman15

Now to check what the user just entered into their config:

inicheck -f config.ini -c master.ini

If you do not get any errors then you have just checked your first config file!

To see how inicheck interpreted your config file add the -w option to the aboved command. This will output the file the way inicheck sees it. Opening it up will show all the entries are filled out.

For more info see Command Line tools.

Aiming to be User Proof¶

For a further demo, let’s say the our user think themselves funny and enters the following into their configuration file:

age: nunya!
job_class: king

inicheck will produce errors and report them in the following manner:

Configuration File Status Report:
==========================================================================
ERRORS:

Section              Item          Message
--------------------------------------------------------------------------
settings             age                Expecting int received str
settings             job_class          Not a valid option

This takes alot of issues out of the hands of the developer so they can focus more on functionality rather than the sometimes erratic behavior of users.

For information take a look at Command Line tools.

For a Project¶

Using inicheck for your project is a great way to save on repeated code and reduce insane amounts of check of types, whether something exists, etc…

Getting a Config File¶

Once the user has filled out a configuration file, inicheck can bring in the information via a UserConfig object by using the following at a minimum:

from inicheck.tools import get_user_config

ucfg = get_user_config(filename, module=str_module_name, checking_later=True)

CAUTION: Using the checking later flag will allow inicheck to pass over issues that would raise exceptions so that the developer can inform the user of the issues with their config file at the same time. This can be dangerous if the developer does not follow up with the functions check_config and print_config_report.

To get the config and check it and report the warnings and errors to the screen:

from inicheck.tools import get_user_config, check_config, print_config_report

ucfg = get_user_config(filename, module=str_module_name, checking_later=True)
warnings, errors = check_config(ucfg)
print_config_report(warnings, errors)

To learn more see checkout the functions documentation:

Installing a Master Configuration File¶

Once you have made a master configuration file, the easiest way to use it for your own project is to add its file path as a module attribute. inicheck currently looks for attributes when a module is requested:

__core_config__

__recipes__

If both are found inicheck will combine them and create a large master configuration file. Below is an example of how to add these attributes to your module.

# Add the following to the packages's __init__.py file
import os
__core_config__ = os.path.abspath(os.path.dirname(__file__) + '/master.ini')

# If you have enough recipes to keep the files separate you can add:
__recipes__ = os.path.abspath(os.path.dirname(__file__) + '/recipes.ini')

Once this is done make sure you add the file(s) to whatever package inclusions you need. Here is an example of how to include them in your setup.py

# In a setup.py, under the setup class initializer
package_data={'mymodule':['./master.ini',
                    './recipes.ini']},

Custom Configuration File Headers¶

inicheck has multiple ways to output your configuration file once inicheck as interpreted it. It is nice to use these because they are clean and it often serves as a placemark on some settings used for some event/task. Developers can provide custom headers. We have seen this functionality used for:

Datetime stamping configuration file creation
Marking with relevant software version numbers or git hashes.
Links to source code or documentation

To add this to your project, simply add:

__config_header__ = some.function()

where “some.function()” would be a custom function that returns a string of content to write to the top of you configuration file.

Here are some examples of this in action:

Custom Configuration File Section Titles¶

A less critical feature but still nice for producing self describing configuration files is having custom section headers.

To add this to your project, simply add the following attribute with a dictionary containing corresponding sections with the messages desired:

__config_titles__ = {"Setup": "This is the setup section"}

The result is a config file with a section preceeded by:

#######################################################################
# This Is The Setup Section
#######################################################################

[setup]
# configuration stuff..

The following is a good example of how this looks in a project:

Master Configuration File¶

Basic Entry¶

To accurately check files, every entry and section must be provided in the master configuration file. Every entry must belong to a section. However, if an option is not registered (in the master config), inicheck will simply produce a warning to alert the user.

Consider the following entry for a configuration file:

[time]

time_step:
          default = 60,
          type = int,
          description = Time interval that SMRF distributes data at in minutes

The example above specifies that the section time has an entry time_step with a default of 60 and should always be an integer. The end outcome is that inicheck requires all provided options for time_step are integers. If the user didn’t provide it, inicheck will add the default.

The attributes describing/constraining an entry are:

default - the default value to use for this entry when not provided

type - the type the value should be casted into, if it is not castable, then throw an error

options - a list of options that the entry value must be in.

description- a description of the entry to be used for documentation and self help

max - A Maximum value used for checking if a numeric input is under it.

min - A minimum value used for checking if a numeric input is over it.

allow_none - A Bool to check if None is an acceptable input

While these are the available attributes they are not required but always exist for each entry. If it is not provided in the master configuration file the following defaults are used.

default - None

type - str

options - None

description- ‘’

max - None

min - None

allow_none - True

if your options are set to none then inicheck assumes the entry can be unconstrained e.g. filenames are a great example where a user would not want a set list of available options.

Available Types¶

Each entry in the master config file can provide a type. If no type is provided, then the default is type string.

Note: Any path options with critical prepended just means that inicheck will throw an error instead of a warning.

Available types:

Bool - CheckBool

String - CheckString

Float - CheckFloat

Integer - CheckInt

Datetime - CheckDatetime

Ordered datetime pair - CheckDatetimeOrderedPair

Filename - CheckFilename

CriticalFilename - CheckCriticalFilename

Discretionary Critical Filename - CheckDiscretionaryCriticalFilename

Directory - CheckDirectory

CriticalDirectory - CheckCriticalDirectory

The following example required the users input to be a string, and must match nearest, linear, or cubic.

[interpolation]
  method:
        default = linear,
        options = [nearest linear cubic],
        description = interpolation method to use for this variable

NOTE ON PATHS: All paths (filenames and directories) in inicheck are assumed to be either relative to the config file or absolute. e.g.

[path_management]
  log:
        default = ../log.txt,
        type = filename,
        description = path to log file

This will default to a path up one directory from the location of the config.

Notes on lists: Listed input checking can be performed. To assign a type as a list, simply add the keyword list to the type name. This will force the output to be a list and still check every entry in a provided list. To provide a list in inicheck master configs use bracketed space separated lists. An example of adding the list keyword and a list default is below

[financial_plots]
  quarterly_dates:
        default = [01-01 04-01 07-01 10-01] ,
        type = datetimelist
        description = List of datetimes to plot up for earnings

Recipes¶

The master config can have recipes to generate certain adding or removing of entries in the users configuration file automatically.This can really help with config file bloat on the users end by only entering the information that matters to them. The the information that matters to the software can be added later by use of recipes.

Recipes are entered like sections but they must have the keyword recipe in the section name. Each recipe is composed of triggers and edits.

Recipe Triggers¶

A trigger describes the conditions for which you want to apply the edits to a configuration file. So you can think of a trigger as a conditional statement. Every trigger is distinguished from edits by having trigger in its entry name. This is required for inicheck to see the triggers!

Triggers can be defined using keywords and to create complex scenarios you can add more keywords. Recipe triggers have only a couple key words available:

has_section - if the configuration file has this section
has_item - It is provded using a bracketed, space delimited list in section > item order
has_value - This is the most flexible trigger provided. It is provded using a bracketed, space delimited list in section > item > value order.

Below is an example showing how a trigger can have multiple criteria that can create very specific conditions. Trigger entries can be provided in a comma separated fashion indicating that the conditions are compounded such that the recipe is applied only if all the entries are true. This allows developers to create highly specific scenarios to apply changes to a users configuration file.

[my_specific_recipe]
       specific_trigger:  has_value = [cool_section cool_item],
                          has_section = test_section,
                          has_value = [super_section awesome_item crazy_value]

If a developer wants more broad conditions to apply changes this can be accomplished by providing another trigger which will be apply a recipe if either trigger is true.

[my_specific_recipe]
       trigger_1:         has_value = [cool_section cool_item]
       trigger_2:         has_section = test_section
       trigger_3:         has_value = [super_section awesome_item crazy_value]

Summary:

Triggers need to have the word trigger in its name
Triggers can compounded by using comma separated values (like an AND statement)
Triggers can be broadened by using multiple triggers (like an OR statement)

Recipe Edits¶

Recipe edits are simply anything in the recipe thats not a trigger and are the edits that will be made to the users configuration file if the recipe is triggered. If a keyword is not used then values are treated like section > item > value and assigned to the users configuration file.

Edits can be prescribed by section and item names under a recipe:

[my_recipe]
  some_trigger:   has_section = test_section

  test_section:
    module = module_name

The example above will assign the value module_name to the item module in the users section test_section if the configuration file has the section test_section

To simplify some entries there are a couple keywords available to reduce repeated actions. Available keywords for entries are:

apply_defaults - apply the defaults set in the section
remove_item - remove an item in this section
remove_section - remove an section in this section
default_item - Applies defaults to all the items provided in a space

delimited list

[topo_ipw_recipe]
trigger_type:       has_value = [topo type ipw]

topo:               type = ipw,
                    apply_defaults = [dem mask],
                    remove_item = filename

Using the entryword default¶

Any item can have the value default given to it which triggers inicheck to look for the default value specified in the master config file and apply during the application of a recipe.

Using the entryword ANY¶

To provide a little more flexibility and again avoid repeated entries, a developer can use the keyword any to capture more generic scenarios. This can be particularly useful when an item is repeated in multiple sections and the developer wants to have the same behavior.

Consider the following configuration file:

[air_temp]
	distribution: dk

[precipitation]
	distribution: dk

Let’s say in the above example we want to have dk_threads everytime a section has distribution: dk. So in the master configuration file we can add a recipe that uses the any keyword to use the same recipe for everytime this happens.

[air_temp]
	distribution:  
		default = idw,
		options = [idw dk]

	dk_threads:
		default = 2,
		type = int

[precipitation]
	distribution:  
		default = idw,
		options = [idw dk]

	dk_threads:
		default = 2,
		type = int

	
[distribution_recipe]

	dk_trigger:
		has_value = [any distribution dk]
	any:
		dk_threads = default	

Recipe Example and Breakdown¶

The following example shows a recipe:

[csv_recipe]
test_trigger:      has_section = csv
csv:          apply_defaults= true
mysql:        remove_section = true
gridded:      remove_section = true

The section name here is identified as a recipe by having the recipe in the name.

2. The trigger is identified by having a name with the word trigger in it and is triggered when the user’s configuration file has a the section csv. 3. If the test_trigger condition is met then all the defaults will be applied to the csv section in the user’s configuration file because of the keyword apply_defaults 4. If the users config file contains the sections mysql and or gridded then they will be removed due to the keywords remove_section

Change Logs¶

If a package is around long enough and is sufficiently configurable, it will almost undergo a refactoring of the configuration file.

To cope with this inicheck allows developers to add a changelog.ini to warn users and remap old config files if need be.

If a changelog is provided, then the inicheck will stop a program if there is a any matches in the changelog that are in the users configuration file.

Usage¶

If your configuration file contains deprecated information as determined by the changelog, then inicheck will exit the program and report the possible issues.

To use inicheck’s CLI to apply the changes simply use

inicheck -f my_config.ini -m my_module --changes -w

This says apply the changes and output the result.

Writing your own change log¶

Syntax¶

Every changelog must have two sections, a meta section and a changes section. The meta section should have an info item with a description of what happened and why. And it should have a date in which indicates the date at which this went into effect.

Future Feature - Currently inicheck will only handle a single changelog but in the future will use the dates to map the changes to each changelog.

A changelog can track the following type of changes:

Removal of a section or item.
The rename of a section or item.
The migrating of an item to another section
Default changes.

An entry in the change log is done with the following format:

<OLD SECTION>/<OLD ITEM> -> <NEW SECTION>/<NEW ITEM>

If declaring a default has changed then the syntax is the same plus the property and value appended.

<OLD SECTION>/<OLD ITEM>/default/<OLD VALUE> -> <NEW SECTION>/<NEW ITEM>/default/<NEW VALUE>

WARNING: If an old default value is identified, auto changing will update any older default values to the new ones.

Look at the following example change log to see various implementations.

Example Changelog

[meta]
info: Major config file changes occurred leading to cleaner files
date: 09-04-2019

[changes]

# Rename an item with in a section
topo/threading -> topo/topo_threading

# Removing an item from a section
topo/dem -> REMOVED

# Removing a section entirely
stations -> REMOVED

# Migrating an item to a new section
solar/distribution -> cloud_factor/distribution

# Migrating an item to a new section and renamed.
solar/slope -> cloud_factor/detrend_slope

# A default value has changed from 0.7 to 1.0
wind/reduction_factor/default/0.7 -> wind/reduction_factor/default/1.0

Where do I keep my change log¶

Change logs can only be associated to a module. Its association is done by creating a module attribute called __config_changelog__.

Example in your <my_module>/__init__.py:

import os
__config_changelog__ = os.path.abspath(os.path.dirname(__file__) + '/path/to/changelog.ini')

Command Line tools¶

inicheck on the command line can greatly speed up debugging of config files and provide simple ways of getting information. In all inicheck CLI tools master config arguments are exchangeable with a python module who has inicheck files installed. For more project related info visit inichecks For a Project.

1. You can access config options by requesting details using the master file or a module name (see For a project).

$ inicheck --details project --master_files examples/master.ini

  Section         Item            Default         Options                   Description
=======================================================================================================================================
project         project_path    None            []                        specifies the project directory path
project         logo_source     None            []                        path to the png for the logo_source
project         website         None            []                        website domain

2. inicheck can also show which recipes were applied using while checking the file.

$ inicheck -f gui_config.ini --master_files examples/master.ini --recipes

Below are the recipes applied to the config file:
Recipes Summary:
================================================================================

user_recipe
--------------------------------------------------------------------------------
    Conditionals:
            trigger_1            settings, user_settings, any
            trigger_2            user_profile, any, any

    Edits:
            settings             autosave             True
            settings             volume               1
            settings             graphics_quality     medium

inicheck can also show which values are non default values.

$ inicheck -f gui_config.ini --master_files examples/master.ini --non-defaults

Configuration File Non-Defaults Report:
The following are all the items that had non-defaults values specified.
============================================================================================================================

Section              Item                 Value                                    Default
----------------------------------------------------------------------------------------------------------------------------
settings             volume               1                                        3
settings             graphics_quality     medium                                   low
settings             user_settings        True                                     False

4. inicheck also has a differencing script for checking how a config file is different from another or different from the master config.

$ inidiff -f ../examples/gui* -mf ../examples/master.ini

Checking the differences...

CFG 1               /home/micahjohnson/projects/inicheck/examples/gui2.ini
CFG 2               /home/micahjohnson/projects/inicheck/examples/gui_config.ini

Section             Item                          CFG 1                         CFG 2                         Default
============================================================================================================================================
settings            volume                        1                             1                             3
settings            graphics_quality              medium                        medium                        low
settings            user_settings                 True                          True                          False
settings            end_test                      2016-10-03 00:00:00           2016-10-01 00:00:00           None

Total items checked: 15
Total Non-default values: 6
Total config mismatches: 1

5. For projects that utilize config files from other projects which may have changelogs, inicheck has a script to find instances of deprecated config items in python files and report the impact. For example, if project A utilizes project B’s config file and project B’s has a changelog. In this scenario you can use the following to search python code in project A’s repo to determine any necessary updates. This could also be used to find changes in the same repo the changelog lives in.

$ inichangefind ./repo_A --modules module_B

Searching ./repo_A for any deprecated config file sections/items in any python files...

Suggested Change                        Affected File                   Line Numbers
=====================================================================================
gridded/n_forecast_hours --> Removed    ./repo_A/framework/framework.py   121
gridded/file --> gridded/wrf_file       ./repo_A/interface.py             189
topo/type --> Removed                   ./repo_A/topo/grid.py             277

inicheck¶

inicheck package¶

Submodules¶

inicheck.changes module¶

class inicheck.changes.ChangeLog(paths=None, mcfg=None, **kwargs)[source]¶

Bases: object

apply_changes(ucfg, potentials, changes)[source]¶

Iterate through the list of detectd applicable changes retrieved from get_actived_changes to produce a new config file with the correct changes.

Parameters:	ucfg – UserConifg Object changes – list lists length 4 (section item property value) representing detected changes
Returns:	Config dictionary
Return type:	cfg

check_log_validity(mcfg)[source]¶: Checks the current master config that the items we’re moving to are valid. Also confirms that removals are aligned with the master.

get_active_changes(ucfg)[source]¶

Goes through the users config looking for matching old configurations, then makes recommendations.

Parameters:	ucfg – UserConfig Object

join_changes(paths)[source]¶

Joins multiple changelogs together

Parameters:	paths – List of paths containing changelogs syntax
Returns:	lists of line item changes provided in each line as a list of [old, new]
Return type:	changes

read_change_log(path)[source]¶

Opens the file and reads in sections and the items.

Parameters:	path – Path to a changlog file

inicheck.checkers module¶

Functions for checking values in a config file and producing errors and warnings

class inicheck.checkers.CheckBool(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Boolean checking whether a value of the right type.

convert_bool(value)[source]¶

Function used to cast values to boolean

Parameters:	value – value(s) to be casted
Returns:	Value returned as a boolean
Return type:	value

class inicheck.checkers.CheckCriticalDirectory(**kwargs)[source]¶

Bases: inicheck.checkers.CheckDirectory

Checks whether a critical directory exists. This is for any directories that have to exist prior to launching some piece of software.

class inicheck.checkers.CheckCriticalFilename(**kwargs)[source]¶

Bases: inicheck.checkers.CheckFilename

Checks whether a critical file exists. This would be any files that absolutely have to exist to avoid crashing the software. These are static files.

class inicheck.checkers.CheckDatetime(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Check values that are declared as type datetime. Parses anything that dateparser can parse.

class inicheck.checkers.CheckDatetimeOrderedPair(**kwargs)[source]¶

Bases: inicheck.checkers.CheckDatetime

Checks to see if start and stop based items are infact in fact ordered in the same section.

Requires keywords section and item to ba passed as keyword args. Looks for keywords start/begin or stop/end in an item name. Then looks for a corresponding match with the opposite name.

start_simulation: 10-01-2019
stop_simulation: 10-01-2017

Would return an issue.

This when checking the start will look for “simulation” with a temporal keyword reference in an item name like end/stop etc. Then it will attempt to determine them to be before.

is_corresponding_valid(value)[source]¶

Looks in the config section for an opposite match for the item.

e.g.: if we are checking start_simulation, then we look for end_simulation

Returns:	item name in the config section corresponding with item being checked
Return type:	corresponding
Raises:	`ValueError` – raises an error if the name contains both sets or None of keywords

is_valid(value)[source]¶

Checks whether it convertable to datetime, then checks for order.

Parameters:	value – Single value to be evaluated
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

class inicheck.checkers.CheckDirectory(**kwargs)[source]¶

Bases: inicheck.checkers.CheckPath

Checks whether a directory exists. These directories are allowed to be none and only warn when they do not exist. E.g. Output folders

class inicheck.checkers.CheckDiscretionaryCriticalFilename(**kwargs)[source]¶

Bases: inicheck.checkers.CheckCriticalFilename

Checks whether a an optional file exists that may change software behavior by being present. In other words, this can be none and still valid. If the not then it registers an error if the string doesn’t exist as path.

class inicheck.checkers.CheckFilename(**kwargs)[source]¶

Bases: inicheck.checkers.CheckPath

Checks whether a directory exists. These are files that the may be created or not necessary to run. E.g. Log files

class inicheck.checkers.CheckFloat(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Float checking whether a value of the right type.

class inicheck.checkers.CheckInt(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Integer checking whether a value of the right type.

convert_to_int(value)[source]¶

When expecting an integer, it is convenient to automatically convert floats to integers (e.g. 6.0 –> 6) but its pertinent to catch when the input has a non-zero decimal and warn user (e.g. avoid 6.5 –> 6)

Parameters:	value – The value to be casted to integer
Returns:	the value converted
Return type:	value

class inicheck.checkers.CheckPassword(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

No checking of any kind here other than avoids alterring it.

class inicheck.checkers.CheckPath(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Checks whether a Path exists. Base for checking if paths exist.

is_valid(value)[source]¶

Checks for existing filename

Parameters:	value – Single value to be evaluated
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

make_abs_from_cfg(value)[source]¶

Looks at a path and determines its absolute path. All paths should be either absolute or relative to the config file in which they will be converted to absolute paths

Parameters:	value – single path or filename
Returns:	absolute path or filename
Return type:	str

class inicheck.checkers.CheckString(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

String checking non paths and passwords. These types of strings are always lower case.

class inicheck.checkers.CheckType(**kwargs)[source]¶

Bases: inicheck.checkers.GenericCheck

Base class for type checking whether a value of the right type. Here extra Attributes are added such as maximum and minimum.

Variables:	minimum – Minimum value for bounded entries. maximum – Maximum value for bounded entries. type_func – Function used to cast the data to the desired type. Default - str() bounded – Boolean indicating if a value can be limited by a min or max.

cast()[source]¶

Attempts to return the casted values of the each value in self.values.

This is performed with self.type_func unless the value is none in which we return None (NoneType)

Returns:	All values from self.values casted correctly
Return type:	list

check()[source]¶

Types are checked differently than some general checker. The checking goes through 5 steps and if anyone of them is invalid it will not check the rest. The process is as follows:

Check if self.values should be a list or not.
Check if a value in self.values should be None or not.
Check for options and if a single value from self.values is among them.
Check is a single value is valid according to self.valid.
Check if a single value is inside any defined bounds.

Returns:	A list equal to the len(self.values) of either None or strings relaying the issues found
Return type:	list

check_bounds(value)[source]¶

Checks the users values to see if its in the bounds specified by the developer.

If self.max or self.min == None then it is assumed no bounds on either end.

Function only runs when bounded = True in the master config.

Parameters:	value – Single value being evaluated against the bounds.
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

check_list()[source]¶

Checks to see if self.values provided are in a list and if they should be.

Returns:	valid - Boolean whether the value was acceptable in terms of being a list msg - string to print if value is not valid.
Return type:	tuple

check_none(value)[source]¶

Check a single value if it is None and whether that is accecptable.

Parameters:	value – single value to be assessed whether none is valid
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

check_options(value)[source]¶

Check to see if the current value being evaluated is also in the list of provided options in the master config. Only runs if options were provided.

Parameters:	value – A single value which is checked against the list of options in the master config
Returns:	valid - Boolean whether the value was in the options list msg - string to print if value is not in the options.
Return type:	tuple

is_valid(value)[source]¶

Checks a single value using the specified type function. All checkers should have a version of this function.

Parameters:	value – Single value to be evaluated
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

class inicheck.checkers.CheckURL(**kwargs)[source]¶

Bases: inicheck.checkers.CheckType

Check URLs to see if it can be connected to.

is_valid(value)[source]¶

Makes a request to the URL to determine the validity.

Parameters:	value – Single value to be evaluated
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

class inicheck.checkers.GenericCheck(**kwargs)[source]¶

Bases: object

Generic Checking class. Every class thats a checker should inherit from this class. This class is used like:

Every check will run the check().

Variables:

message – String message to report if the value passed is not valid
msg_level – Urgency of the message which can be either a warning or error
values – value to be checked, casted, and reported on
config – UserConfig object that the item/value being check resides
type – string name representing the datatype which is based off the class name
is_list – Boolean specifying the resulting values as a list or not
allow_none – Boolean specifying the values can contain a value
E.G. –
b = GenericCheck(section=’test’, item= ‘tiem1’, config=ucfg) issue = b.check() if issue != None:

log.info(b.message)

check()[source]¶

Function that is ran by the checking function of the user config to return useful message to instruct user.

Returns:	None if the entry is valid, else returns self.message
Return type:	msgs

is_it_a_lst(values)[source]¶

This checks to see if the original entry was a list or not. So instead of evaluating self.values which is always a single item, we evaluate self.config[self.section][self.item]

Parameters:	values – The uncasted entry from a config file section and item, can be a list or a single item
Returns:	True if its a list false if it is not.
Return type:	boolean

is_valid(value)[source]¶

Abstract function for defining how a value is checked for validity. It should always be used in check(). Is valid should always return a boolean whether the result is valid, and the issue as a string stating the problem. If no issue it should still return None.

Parameters:	value – Single value to be evaluated
Returns:	valid - Boolean whether the value was acceptable msg - string to print if value is not valid.
Return type:	tuple

inicheck.cli module¶

inicheck.cli.check_for_changes(directory, modules=[], paths=[])[source]¶

Returns files with changes found in the changelog. Using the change log this function goes through all the files in the directory and returns any filenames with a config entry in python code that will error out

This script only looks at and reports changes in found for sections and items listed in the changelog. Currently ignores default changes.

Parameters:	directory – Folder with python scripts to check for instances of changes paths – list of paths to an inicheck changelog modules – list of python modules with an __config_changelog__ attribute
Returns:	dictionary with filenames as keys and line numbers in a list as the value.
Return type:	change_instances

inicheck.cli.detect_file_changes()[source]¶: CLI tool for examining a python repo and reporting any old python code that references a deprecated config file entry as reported in a changelog

inicheck.cli.inicheck_main(config_file=None, master=None, modules=None, write_out=False, show_recipes=False, show_non_defaults=False, details=None, apply_changelog=False)[source]¶: Function used for the CLI for inicheck. This is mostly for cleaner testing. Allows users to look at master config details, check their config file against the master, look at recipes applied, examine non-default entries, write new config files, and apply changes from a changelog

inicheck.cli.inidiff()[source]¶: Creates a report showing the difference in files

inicheck.cli.inidiff_main(config_files, master=None, modules=None)[source]¶

inicheck.cli.inimake()[source]¶

Attempts to walk through making a brand new ini file based on developers master config and recipes.

Currently config files can have different configurations based on recipes. This script will ask the user for a decision by looking for:

Triggers built on sections and if that section is removed in another conditional
Triggers built on any_section item trigger which is also removed in another conditional

inicheck.cli.main()[source]¶

inicheck.config module¶

class inicheck.config.MasterConfig(path=None, modules=None, checkers=None, titles=None, header=None, changelogs=None)[source]¶

Bases: object

add_files(paths)[source]¶

Designed to add to the master config file if the user has split up files to reduce the amount of info in a master file. e.g. recipes are stored in another file.

Parameters:	paths – list of real path to another cfg.ini
Returns:	Original config with appended information found in the extra cfg
Return type:	config

merge(mcfg)[source]¶: Merges the a master config object into the current master config object in place

class inicheck.config.UserConfig(filename, mcfg=None)[source]¶

Bases: object

Class meant for managing the the users config, here we operate on the config repeatedly making it available through the attribute self.cfg

Variables:

raw_cfg – Untouched original OrderedDict that inicheck read from file
cfg – OrderedDict of the config file that inicheck will check, cast, list, etc
recipes – List of entries.recipes.RecipesSection that apply to this config
sections – List of strings that represent the unique sections for the whole config file
items – List of strings that represent the unique items for the whole config file
values – List of strings that represent the unique values for the whole config file
mcfg – config.MasterConfig object that represents the standard the cfg is checked against

add_defaults(cfg, sections=None, items=None)[source]¶

Look through the users config file and section by section add in missing parameters to add defaults

Parameters:	sections – Single section name or a list of sections to apply (optional) otherwise uses all sections in users config
Returns:	User config dictionary with defaults added.
Return type:	user_cfg

apply_recipes()[source]¶

Look through the users config file and section by section add in missing parameters to add defaults

Returns:	User config dictionary with defaults added.
Return type:	user_cfg

get_unique_entries(cfg)[source]¶

Appends all the values in the user config to respectives lists of section names, item names, and values. Afterwards any copy is removed so all is left is a unique list of names and values

Parameters:	cfg – OrderedDict of the config file
Returns:	tuple of len=3 of sets of unique sections items and values
Return type:	tuple

interpret_recipes(partial_cfg, situation)[source]¶

User inserts a partial config by using each situation that triggered a recipe. A triggering situation consists of a tuple of (section, item, value) that represent the specific settings that trigger the recipe. All default references should avoid overwriting the users selection.

Parameters:	partial_cfg – dictionary of edits to be applied to the cfg situation – List of len=3 describing the trigger mechanism
Returns:	Modified dictionary
Return type:	result

update_config_paths(user_cfg_path=None)[source]¶: Sets all paths so that they are always relative to the config file or absolute.

inicheck.config.check_types(cfg, checkers)[source]¶

Iterates through all the master config items and confirm all type are valid

Parameters:	cfg – dict of master config entries listing out properties checkers – dict of checker names and class as Key Value pairs
Returns:	True if no error is raised
Return type:	Bool
Raises:	`ValueError` – raises error if a type is specified not in the lists of checkers

inicheck.entries module¶

class inicheck.entries.ConfigEntry(name=None, parseable_line=None)[source]¶

Bases: object

ConfigEntry designed to aid in parsing master config file entries. This is meant to parse:

Example:

------------------------------------------------------------
item:
                type = <value>,
                options = [<value> <value>],
                description = text describing entry
-------------------------------------------------------------

Config entry expects to recieve the above in the following format:

parseable_line = ["type = <value>",
                 "options = [<value> <value>"],
                 "description=text describing entry"]
name = item

Config entry then will parse the strings looking for space separated lists, values denoted with =, and will only receive:

type

default

options

description

max

min

allow_none

class inicheck.entries.RecipeSection(recipe_section_dict, name=None)[source]¶

Bases: object

The RecipeSection enables the master config file to parses a section that is actually a recipe. Operates using keywords provided n the name of the section. It collects triggers and adjustments to the config file and assigns them to this class. Trigger represent the conditional statements to apply the adjustments listed below. If an item in the section doesn’t contain any trigger keyword then it is assumed to be an adjustment to be made.

class inicheck.entries.TriggerEntry(parseable_line, name=None)[source]¶

Bases: object

RecipeEntry designed to aid in parsing master config file entries under a recipe. This is meant to parse:

Example:

------------------------------------------------------------
item_trigger:
                has_section_name = <value>,
                has_value = [<section name> <item name>, <value>],
                has_item_name = <value>

-------------------------------------------------------------

Config entry expects to recieve the above in the following format:

{item_trigger:
            ["has_section = <value>",
             "has_item = [<section> <item>"],
             "has_value = [<section> <item> <value>]"
             ]
}

Recipe entry then will parse the strings looking for space separated lists, values denoted with = and will only accept keyword

has_section

has_item

has_value

inicheck.iniparse module¶

inicheck.iniparse.parse_changes(change_list)[source]¶

Takes in a section and item parsed dictionary and parses on > forming a list. It works as the following:

To move an item to another section or item name section/item -> new_section/new_item

section/item -> Removed

Parameters:	change_list – List of modifcations that occured to the config
Returns:	a list of a list length 2 containing of list length 4 representing section item property value which are all any by default.
Return type:	result

inicheck.iniparse.parse_entry(info, item=None, valid_names=None)[source]¶

Parse the values found in the master config where the entries are a little more complicated. Specifically this deals with syntax issues and whether a entry option is valid or not.

This should be used only after knowning the section and item.

Parameters:	info – a single line or a List of lines to parse containing master config options item – Item name were parsing. Only for reporting errors purposes valid_names – valid property names to be parsing.
Returns:	dictionary containing the properties as keys
Return type:	properties

inicheck.iniparse.parse_items(parsed_sections_dict, mcfg=None)[source]¶

Takes the output from parse_sections and parses the items in each section

Parameters:	parsed_sections_dict – dictionary containing keys as the sections and values as a continuous string of the
Returns:	dictionary of dictionaries containing sections,items, and their values
Return type:	result

inicheck.iniparse.parse_sections(lines)[source]¶

Returns a dictionary containing all the sections as keys with a single string of the contents after the section

Parameters:	lines – a list of string lines containing all the raw info of a cfg file
Returns:	dictionary containing keys that are the section names and values that are strings of the contents between sections
Return type:	sections

inicheck.iniparse.parse_values(parsed_items)[source]¶

Takes the output from parse_items and parses any values or properties provided in each item placing the strings into a list. If no properties are defined then it will clean up values provided and put them in a list of len one.

Parameters:	parsed_sections_dict – dict of dicts containing joined strings found under each item
Returns:	dictionary of dictionaries containing sections, items, and the values provided as a list
Return type:	result

inicheck.iniparse.read_config(fname)[source]¶

Opens and reads in the config file in its most raw form. Creates a dictionary of dictionaries that contain the string result

Parameters:	fname – Real path to the config file to be opened
Returns:	dict of dicts containing the info in a config file
Return type:	config

inicheck.output module¶

inicheck.output.generate_config(config_obj, fname, cli=False)[source]¶

Generates a list of strings using the config data then its written to an .ini file

Parameters:	config_obj – config object containing data to be outputted fname – String path to the output location for the new config file cli – Boolean value that adds the line “file generated using inicheck.cli”, Default = False

inicheck.output.print_cfg_for_recipe(cfg, fmt, hdr=None)[source]¶

inicheck.output.print_change_report(potential_changes, required_changes, ucfg, logger=None)[source]¶

Pass in the list of changes generated by check_config file. print out in a pretty format the changes required

Parameters:	potential_changes – List of warnings about config property changes especialy about defaults retruned from `get_active_changes()`. potential_required – List of critical changes returned from `get_active_changes()`.

inicheck.output.print_config_report(warnings, errors, logger=None)[source]¶

Pass in the list of string messages generated by check_config file. print out in a pretty format the issues

Parameters:	warnings – List of non-critical messages returned from `check_config()`. errors – List of critical messages returned from `check_config()`. logger – pass in the logger function being used. If no logger is provided, print is used. Default = None

inicheck.output.print_details(details, mcfg)[source]¶

Prints out the details for a list of provided options designed for use with the CLI. Details about a section, or an item can be requested by passing in a list of in the section,item order. If a section is only passed then we the details provided are for the entire section

Parameters:	details – a list in [section item value] requesting details. mcfg – master config dictionary to gather the details from

inicheck.output.print_non_defaults(ucfg)[source]¶

Prints out the options used that were not default option values.

Parameters:	ucfg – config object containing options that are not default

inicheck.output.print_recipe_summary(lst_recipes)[source]¶

Prints out the recipes found and how they are interpretted

Parameters:	lst_recipes – list of the recipe entry objects

inicheck.tools module¶

inicheck.tools.cast_all_variables(config_obj, mcfg_obj)[source]¶

Cast all values into the appropiate type using checkers, other_types and the master config.

Parameters:	config_obj – The object of the user config from mcfg_obj – The object used for manage the master config from class MasterConfig other_types – User provided list to add any custom types
Returns:	The users config dictionary containing the correct value types
Return type:	ucfg

inicheck.tools.check_config(config_obj)[source]¶

Looks at the users provided config file and checks it to a master config file looking at correctness and missing info.

Parameters:	config_obj - UserConfig object produced by – `UserConfig`
Returns:	warnings - Returns a list of string messages that are consider non-critical issues with config file. errors - Returns a list of string messages that are consider critical issues with the config file.
Return type:	tuple

inicheck.tools.config_documentation(out_f, paths=None, modules=None, section_link_dict={})[source]¶

Auto documents the core config file. Outputs to a file which is can then be used for documentation. Specifically formulated for sphinx

Parameters:	out_f – string path to output location for the auto documentation paths – paths to master config files to use for creating docs modules – modules with attributes __core_config__ for creating docs section_link_dict – dictionary containing special documentation for a section in the config file reference

inicheck.tools.get_checkers(module='inicheck.checkers', keywords='check', ignore=['type', 'generic', 'path'])[source]¶

Parameters:	module – The module to search for the classes keyword – Keywords to look for in class names ignore – Post parsed keywords to ignore if found in a class name
Returns:	Dict of the classes available for checking config entries
Return type:	dictionary

inicheck.tools.get_merged_checkers(ucfg)[source]¶

Retrieve the dictionary of checker classes by grabbing all in inicheck and any prescribed in the master config through another module

Parameters:	ucfg – User Config object containing modules
Returns:	all_checks - dictionary of all the checkers from inicheck and any modules assigned to the master config.
Return type:	dictionary

inicheck.tools.get_user_config(config_file, master_files=None, modules=None, mcfg=None, changelog_file=None, cli=False)[source]¶

Returns the users config as the object UserConfig.

Parameters:	config_file – real path to existing config file master_file – real path to a Core Config file modules – a module or list of modules with a string attribute __CoreConfig__ which is the path to a CoreConfig mcfg – the master config object after it has been read in. changelog_file – Path to a changlog showing any changes to the config file the developers have made cli – boolean determining whether to attempt to process the changes which should be done from the CLI only
Returns:	Users config as an object
Return type:	ucfg

inicheck.utilities module¶

inicheck.utilities.ask_config_setup(choices_series, section=None, item=None, num_questions=None)[source]¶: Ask repeating questions for setting up a config

inicheck.utilities.ask_user(question, choices=None)[source]¶: Asks the user which choice to make and handles incorrect choices.

inicheck.utilities.find_options_in_recipes(recipes, choice_search, action_kw, condition_position=0)[source]¶

Looks through the master config at recipes and entries to determine if there are place the developer made distince choices, this is used in the inimake script to walk users through making their own config file from scratch.

This function looks through and finds sections or items that match the conditional. Then it looks through the adj_config to determine if any sections/items are being removed as a result, then it finds those. If a section/item appears in a recipe condition and a removed by another trigger it is extremely likely these are choices.

Parameters:	mcfg – Recipes object from master config conditional – list of 3 focusing on what were looking for action_kw – action keyword is a string of what to look for in matching choices where ever true is used will popoulate with every item/section
Returns:
Return type:	final_choices

inicheck.utilities.get_inicheck_cmd(config_file, modules=None, master_files=None)[source]¶: Strings together an inicheck cli command based on modules and files

inicheck.utilities.get_kw_match(potential_matches, kw_list, kw_count=1)[source]¶

Loops through a list of potential matches looking for keywords in the list of strings being presented. When a match is found return its value.

Parameters:	potential_matches – List of strings to check kw_list – List of strings to look for inside the single word kw_count – Minimum Number of keywords to be found for it to be True
Returns:	The first potential found with the keyword match
Return type:	result

inicheck.utilities.get_relative_to_cfg(path, user_cfg_path)[source]¶

Converts a path so that all paths are relative to the config file

Parameters:	path – relative str path to be converted user_cfg_path – path to the users config file
Returns:	absolute path of the input considering it was relative to the cfg
Return type:	path

inicheck.utilities.is_kw_matched(single_word, kw_list, kw_count=1)[source]¶

Checks to see if there are any keywords in the single word and returns a boolean

Parameters:	single_word – String to be examined for keywords kw_list – List of strings to look for inside the single word kw_count – Minimum Number of keywords to be found for it to be True
Returns:	Whether a keyword match was found
Return type:	boolean

inicheck.utilities.is_valid(value, cast_fn, expected_data_type, allow_none=False)[source]¶

Checks whether a value can be converted using the cast_fn function.

Parameters:	value – Value to be considered cast_fn – Function used to determine the validity, should throw an exception if it cannot expected_data_type – string name of the expected data allow_none – Boolean determining if none is valid
Returns:	valid (boolean): whether it could be casted msg (string): Msg reporting what happen
Return type:	tuple

inicheck.utilities.mk_lst(values, unlst=False)[source]¶: while iterating through several type of lists and items it is convenient to assume that we recieve a list, use this function to accomplish this. It also convenient to be able to return the original type after were done with it.

inicheck.utilities.parse_date(value)[source]¶

Function used to cast items to datetime from string. Meant to prevent the importing of pandas just for the to_datetime function.

Parameters:	value – string of a datetime
Returns:	Datetime object representing the value passed.
Return type:	converted

inicheck.utilities.pcfg(cfg)[source]¶

prints out the config file to the prompt with a nice stagger Look for readability

Parameters:	cfg – dict of dict in a heirarcy of {section, {items, values}}

inicheck.utilities.pmcfg(cfg)[source]¶

prints out the core config file to the prompt with a nice stagger Look for readability

Parameters:	cfg – dict of dict in a hierarchy of {section, {items, values}}

inicheck.utilities.remove_chars(orig_str, char_str, replace_str=None)[source]¶

Removes all character in orig_str that are in char_str

Parameters:	orig_str – Original string needing cleaning char_str – String of characters to be removed replace_str – replace values with a character if requested
Returns:	orig_str with out any characters in char_str
Return type:	string

inicheck.utilities.remove_comment(string_entry)[source]¶: Takes a single line and removes and .ini type comments. Also remove any spaces at the begining or end of a commented line

Module contents¶

Contributing¶

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

You can contribute in many ways:

Types of Contributions¶

Report Bugs¶

Report bugs at https://github.com/USDA-ARS-NWRC/inicheck/issues.

If you are reporting a bug, please include:

Your operating system name and version.
Any details about your local setup that might be helpful in troubleshooting.
Detailed steps to reproduce the bug.

Fix Bugs¶

Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.

Implement Features¶

Look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.

Write Documentation¶

inicheck could always use more documentation, whether as part of the official inicheck docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback¶

The best way to send feedback is to file an issue at https://github.com/USDA-ARS-NWRC/inicheck/issues.

If you are proposing a feature:

Explain in detail how it would work.
Keep the scope as narrow as possible, to make it easier to implement.
Remember that this is a volunteer-driven project, and that contributions are welcome :)

Get Started!¶

Ready to contribute? Here’s how to set up inicheck for local development.

Fork the inicheck repo on GitHub.

Clone your fork locally:

$ git clone git@github.com:your_name_here/inicheck.git

Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:
```
$ mkvirtualenv inicheck
$ cd inicheck/
$ python setup.py develop
```
Create a branch for local development:
```
$ git checkout -b name-of-your-bugfix-or-feature
```
Now you can make your changes locally.
When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
```
$ flake8 inicheck tests
$ python setup.py test or py.test
$ tox
```
To get flake8 and tox, just pip install them into your virtualenv.

Commit your changes and push your branch to GitHub:

$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature

Submit a pull request through the GitHub website.

Pull Request Guidelines¶

Before you submit a pull request, check that it meets these guidelines:

The pull request should include tests.
If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
The pull request should work for Python 3.5 and 3.6 and for PyPy. Check https://github.com/USDA-ARS-NWRC/inicheck/pull_requests and make sure that the tests pass for all supported Python versions.

Tips¶

To run a subset of tests:

$ py.test tests.test_inicheck

Credits¶

Development Lead¶

Micah Johnson <micah.johnson150@gmail.com> @micahjohnson150

Contributors¶

Micah Sandusky <micah.sandusky@ars.usda.gov> @micah-prime

History¶

0.1.0 (2018-01-05)¶

First release on PyPI.

0.2.0 (2018-04-11)¶

Recipes were added in
Custom Checkers
Multiple master files can be used
Module associated files to find master configs
Detail finding in cli to get help on the fly

0.3.0 (2018-08-10)¶

Added the ability require an config entry be a list
Added significant unit tests

0.4.0 (2019-07-19)¶

Added in url datatype
Added Datetime Ordered data type for start stop options
Added in an inidiff script for comparing config files

0.5.0 (2019-08-22)¶

Fixed bug relating to Datetime Ordered Pairs
Added in inimake script for creating scripts from scratch
Added in more unittests (54% coverage)

0.6.0 (2019-10-16)¶

Built in a brand new feature for migrating large changes in configs
Added bounds checking for type checking allowing for continuous types
Fixes for issues: 25 , 27 , 31 , 32

0.7.0 (2019-10-22)¶

Restructure the checkers some to better handle types
Fixed several broken tests.

0.8.0 (TBD)¶

Increased unittest coverage (57%-83%)
Added in a CLI tool to find changelog impacts in python files
Switch from pandas to_datetime to dateparser to reduce package size
Fixes for 35 , 38 , 39, 40, 41, 42 43, 44, 45

Welcome to inicheck’s documentation!¶