Configargparser Usage

How to Use Configargparser

Introduction

In this blog, I want to show you somethings about how to use the “Configargparser” module to realize some basic config for your procedures. Configargparser is a more stronger module, but the relevant document is very few.

There are three main ways that programs get data from their environment when they start up:

  • Commandline arguments,
  • Configuration files
  • Environment variables

Currently, if a python program wants to parse data from these sources, it needs to first check the environment variables, then check for a configuration file and finally look at the commandline arguments. The general cascade is that commandline arguments should override config files which should override environment variables. Unfortunately, checking all of these places is a major pain – which is what configargparser is meant to address.

It’s known that Python has provided two modules, argparser for commandline arguments parsing and configparser for configuration files. While Configparser can address those in a single module.

Install

If Internet is available, you can install Configargparse with:

1
pip install configargparser

If not, you can download the configargparser package at:

https://pypi.org/project/ConfigArgParse/

and then install from source by:

1
python setup.py install

By the way, the repo on Github is also available at:

https://github.com/mgilson/configargparser

Basic Usage

If you are similar with argparse, you can skip this section, because configargparser use the same way as argparser.

1
2
3
4
5
6
7
8
import configargparse

p = configargparse.ArgumentParser()
p.add('-v',help='verbose',action='store_true')
p.add('-s','--size', type=int, choices=[5, 10])
opt = p.parse_args()
print(opt)
print('size is %d'%opt.size)

Run the file by:

1
python test.py -v -s 10

You can see:

1
2
Namespace(size=10, v=True)
size is 10

From Config File

We can also load the config parameters from config file, a .ini file or .yml file. For example, we first prepare a test.yml file with a key: value format:

1
2
3
data: /home/daizelin
size: 10
embedding_dim: 512
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import configargparse

p = configargparse.ArgumentParser(default_config_files=['/home/zelin/parser_test/test.yml'],config_file_parser_class=configargparse.YAMLConfigFileParser)
# You can also specify the config file path with a -c parameter if the following code works.
## p.add('-c','--custom-config', required=True, is_config_file=True,help='config_file_path')
p.add_argument('-data','--data', help='data')
p.add_argument('-size','--size', type=int, help='size')
p.add_argument('-embedding_dim','--embedding_dim', type=int,help='embedding_dim')
opt = p.parse_args()
print(opt)

print('-'*68)
print(p.format_help())
print('-'*68)
print(p.format_values())

Run the file by:

1
python test.py

You can see:

1
Namespace(data='/home/daizelin', embedding_dim=512, size=10)

Run the file by:

1
python test.py -embedding_dim 128

You can see:

1
Namespace(data='/home/daizelin', embedding_dim=128, size=10)

This means that the command line arguments will overwrite the config file arguments.

Some tips:

  • The parameters in the yml file adapt the format as key: value. Note the whitespace after :

  • When reading from a config file, you need also add the arguments to the parser by:

    1
    p.add_argument('-data','--data', help='data')

Group

For clarity, we usually use argument group for different codes.

1
2
3
4
5
6
7
8
9
10
import configargparse

p = configargparse.ArgumentParser()
group_1 = p.add_argument_group("Model")
group_1.add('--batch_size', type=int, default=64)
group_1.add('--learning_rate', type=float, default=0.001)
group_2 = p.add_argument_group("Preprocessing")
group_2.add('--max_len', type=int, default=50)
opt = p.parse_args()
print(opt)

Run the file by:

1
python group.py

You will see:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Namespace(batch_size=64, learning_rate=0.001, max_len=50)
----------------------------------------------
usage: group.py [-h][--batch_size BATCH_SIZE] [--learning_rate LEARNING_RATE][--max_len MAX_LEN]

optional arguments:
-h, --help show this help message and exit

Model:
--batch_size BATCH_SIZE
--learning_rate LEARNING_RATE

Preprocessing:
--max_len MAX_LEN

-----------------------------------------------

Defaults:
--batch_size: 64
--learning_rate: 0.001
--max_len: 50