complete rewrite of the difference converter system that uses
deepops.deepdiff() to work out differences more simply, based on the
configuration dictionaries directly

rule system replaced by simpler 'excludes' list which matches keys in
the configuration dictionaries: -x/-X/-y replace old options

add various debugging options: -P, -C, -D, -V

added -O option to suppress output

was using 'addr' instead of 'grp' for the HSRP group number

commented-out 'interface / cdp enable', and 'standby' converters as
these are not present in the previous master version and are finding
differences on the live network: they will need to be fixed up first

split the 'interface / shutdown' converter into one to do 'shutdown'
and the other 'no shutdown', positioned at the start and end of the
block of interface converters: the rationale is that the interface
should be shut down before any configuration changes are made, but
enabled after they are changed, to avoid disruption

fixed a bug where 'no spanning-tree' conversions were reversed (since
removing 'no spanning-tree' actually enables it, and vice-versa)

did a comparison with the live network configuration and the current
state results in the same conversions (albeit in a different order)
as that, so this version could be swapped in

this is used to add the default excludes for particular platform - for
IOS, it excludes the default system CoPP ACLs

also, tidied some comments in iosdiff

also, renamed ConfigDiff to DiffConfig to better fit the function of
the class

renamed the do_conversion() method to convert()

moved the explain and debug parameters which were passed to the do_-
conversion() method to be parameters passed to the constructor and
stored in the object

DiffConfig._add_converter() now takes a DiffConvert object and not a
class, so the caller must instantiate it themselves - this seems more
'correct'

minor other tidying and removing of extraneous whitespace

believe this is now in a state to replace the previous converter in
the master branch

reviewed and tidied the comments in that class and renamed it to a more
appropriate name

previously only dictionaries could be matched with wildcards, forcing
a converter wanting to work through differences in a set to match the
level above and iterate across the set - now they can be matched
directly

this is useful for matching static route changes (for example)

it may still be desirable to do the old method, if the commands to be
added are all in a subcontext, to avoid lots of extraneous commands
entering the subcontext (e.g. 'interface X', 'switchport trunk allowed
vlan add Y' only having one 'interface X' for the block of VLAN
changes)

they did use lists: tuples are slightly cleaner to write as you don't
need to put square brackets around them (although you do need to follow
a single item was a comma to indicate it's not a simple value)

folded the 'excludes' functionality that was in configrules into the
configdiff module (ConfigDiff class) - as this has been significantly
simplified, there was very little left and it didn't need to be a
separate class (and, thus, file)

the configchanges file handled the blocked configuration change
mechanism (which started and stopped blocks of contextual configuration
changes, disposing of them if there were no changes within) as this is
now all defunct with the new converter mechanism

the main program has been rewritten to accommodate some of these
changes

other changes:

when debugging the conversion process, paths being checked for in the
difference dictionaries now show '*' for a wildcard field, rather than
the actual value in the match path of None

final clearing out before looking to fold this into ConfigDiff

replacing pathstr() now there's a function in deepops 1.6.1

this is the same function, but it's now a standard part of the deepops
module so we use that one instead

this removes all the code which has been replaced by the new parser
and converter system, including all the old blocked configuration
change system and match rules

the code still needs to be tidied and commented correctly and a few
references fixed up, but all the code should now be the new system

this shows the path which matches a converter above each block of
generated commands

the old rule system (which broke when the new converters were
introduced) has been replaced by the 'exclude items' system, which is
much simpler and just contains paths through the configuration
dictionaries to be excluded from the comparison

this new system is much simpler and doesn't involve manually having to
check for rules at each change

the command line switches for these have been changed to match the name
change; a -X option has been added to debug the exclude items by
dumping them out, before running

other changes:

the -O option has been fixed (it previous didn't disable output
properly)

minor corrections and code tidying

as per IPv4 access-lists and prefix-lists

these were in the old differences calculator so need to be added,
before the old code can be removed

work is still in progress but things are now working so this seems like
a good point to stop and checkpoint everything

the old process of manually working through the 'from' and 'to'
configurations has been replaced by a new system with makes use of the
deepops.deepdiff() function which takes two dictionaries and compares
them, returning a 'remove' and an 'update' dictionary

the diffing process consists of a number of 'converters' which match
elements in the difference dictionaries to trigger remove() and
update() methods on those elements

for example, if config["hostname"] appears in the remove dictionary,
the hostname has been removed from the configuration; similarly, if it
appears in the update dictionary, the hostname has either been added
or changed

converters can also match wildcard paths into the dictionaries, which
will then expand into all matches - for example config["vlan"][*]-
["name"] will match any VLAN where the name has been removed or updated
and the methods will be called for each one, with the wildcard parts of
the path supplied to the method - multiple levels can be wildcarded and
all the expansions matched

this change makes the conversion process much simpler and avoids
getting tied up in knots of checking levels of differences in the
dictionaries and handling multiple levels where the are multiple
matches to iterate through much easier - for example, handling
'interface ... standby ... ip ... secondary' was previously very
difficult but is trivially handled here

no other extra diffs are handled, aside from HSRP addresses, to test
the new mechanism was fit for purpose: it should be significantly
easier to add new things now

other changes:

added the debugging options -P, -C, -D and -V to replace the single
-D option; -V supports multiple instances to increase verbosity

added the -O option to suppress normal output (helpful when using
debugging options)

things outstanding that need to be fixed before final release:

the rule system is currently completely broken (it's not called in any
of the new conversion code) and needs to be reimplemented - the plan is
to do this by pruning things out of the difference dictionaries, which
will avoid a separate matching process

the old code to handle the diffing and conversions is currently all
still there as it was being referred to, during the rewrite - when
everything is working, this will be removed

similarly, the new code is a little untidy (although all commented and
believed to be working correctly): this will be done when the old code
is removed

built-in default ACLs (typically the "acl-copp-..." ones) are diffed
and really should be excluded, when using configurations from the
generator, as they wouldn't typically be there

the matcher for conversions should really match things other than
dictionaries with wildcards - lists and sets - this would make
handling some things (like "ip route ...") much easier as it would
avoid having to explicitly iterate through the differences

ACLs used to be diffed using a function which tried to explain how the
lists had changed (by highlighting places where they differ and are the
same) - this is difficult to do using the new mechanism as the old and
new lists are not automatically available; for now, the old list is
just cleared, then shown, commented out, before the new list is loaded;
the old mechanism may be reinstated in future, if deemed necessary (and
a good way of doing it has been thought of)

this is useful when initialising a path of dictionaries using
setdefault(), as done regularly in the parser action methods

this (and setdefault()) also return the final object in the path and so
this is used to make the action functions a little smaller and more
efficient

if no 'to' argument is specified, only the 'from' file will be read in
and parsed, with debugging mode enabled by default

the basic parsing algorithm and routines are the same but the
implementation has been overhauled to simplify it and improve
performance

the ~ConfigParser classes are now renamed ~Config as they really
represent the configuration (with parsing being the main mechanism to
get the configuration into it)

the [now] config classes are also descendants of the dict class, so can
be queried directly, removing the get_config() method

a configuration file can be parsed upon instantiation by passing the
filename to the constructor

commands are now parsed using IntendedContextualCommand class - this
wraps up the context, command regular expression, action function
and if it enters a subcontext into a class, avoiding the need for this
to be scattered about (in particular separating the adding of it to
config oject from the definition, which was commonly missed out)

subcontext action methods get a more specific element of the
configuration dictionary by the action function returning it, rather
than having to add it as a 'config' item to the 'params' dictionary

the full configuration is no longer supplied to the action method

an action function called in the top level context will receive the
entire configuration dictionary by default

the IOS parser has been reorganised to make it simpler - it builds a
list of command classes to add to the config object which makes it
easier to read and edit

added some functions to perform common activities (such as taking a
string list of numbers with ranges and turning it into an expanded set
of numbers)

tidied comments

lots of little tidies and optimisations and now the code runs
approximately just over twice as fast as the previous version on a set
of configurations files (around 4s instead of 9s for ~140k lines of
source configurations and ~520k lines of new configuration)

the parsing was changed in 0.7.3 and a bug was introduced where missing
options would be filled in with the string 'None' - this would only
matter if there were differences and this would be included in the
changed lines

the 'log' option to ACEs was parsed but not included in differences

the package now names 'netaddr' as a dependency

fixed a bug where debugging parsing would crash due to non-inclusion
of module (sys)

clearer messages explaining parsing of configuration file

fixed Makefile to not use isolated environment (to allow twine to be
run from user modules)

the rename from cfg_contextdiff to net_contextdiff involved changes in
more places than originally made - made those

this also corresponds to the first upload to the new PyPi module
repository

to make the project name format more consistent

now calculates differences between 'switchport trunk allowed vlan ...'
and 'switchport trunk native vlan ...' configurations

also, differences for 'cdp enable'

fixed bug where ACL rules (and their explained comments) were incorrect
(ordering of arguments to function wrong, again!)

removed versions from individual module files

interfaces were deleted without checking the rules, nor was an explain
comment added

indentbegin() and indentend() methods added to allow blocks of appended
text to be indented by the specified string using a stack of indent
strings

beginbegin() and beginend() have also been extended to support starting
and ending an indented section as part of the block, with an optional
leading line (at the beginning of the block) that is not indented; this
allows managing the indent to be combined with the block methods

IOS code for VLANs and interfaces updated to use this functionality:
this results in the comments added using the 'explain includes' option
to have the appropriate indenting (previously they were also flush left
as the method didn't know how indented the configuration currently was)

script would crash if an interface was to be deleted

interfaces are now no longer deleted if they are 'fixed' interfaces
(physical interfaces which may not exist in the new configuration as
they're unconfigured, but mustn't be removed) - interfaces such as
VLANs and port-channels are still removed

fixed some bugs where the order of elements in a rule would be muddled
up (e.g. the area used as the operation) - this was because there was a
mix of different orders to method parameter lists: these are now always
quoted in op-area-id order to match that in the string form of a rule

now handles interface aliases for VLANs

now computes differences for IP and IPv6 addresses on interfaces

also, some underlying reorganisation of code, including:

* moved comparison methods from the ConfigElementRules class to the
  ConfigDiff class

* added the _compare_set_include() method to ConfigDiff to make
  comparing sets (of address, routes, etc.) easier and consistent

* the compare_..._matchinfo() variants of methods are now the base (non
  ..._matchinfo()) versions - if just the 'include' flag is needed,
  this can be extracted with ...()[0] or other and avoids maintaining
  two largely identical sets of methods downstream

a number of devices can be specified on the command line to be
processed, one after the other

output can now be sent to a file instead of just stdout (and needing to
be redirected by the caller)

if this function is used, the 'from filename', 'to filename' and
'output filename' arguments have '%' replaced with the name of the
device being processed

also, some minor tidying and bugfixes

changed the format of rules to move the 'operation' field before the
'area' and 'id' fields - i.e. '[!][{+-=}][<area>][:<id>]' replaces the
previous '[!][<area>][:<id>][{+-=}]'

as well as making the rule syntax clearer, this change fixes a problem
where a '-' could not be used in the 'action' field - previously this
was not allowed as the regexp for that did not match '-' but, when it
was added, caused the regexp to match it as an 'operation'

moved the code to read in rules from a file from the main module to the
the ConfigElementRules class as read_rules(), extending it to support
comments in files with a leading '#' and skip blank lines

the script is not finished but this repository has been created
following the packaging of it

0.3 was picked as there were a series of alpha versions used, prior to
packaging, ending at 0.23