MP Bulk Configuration tool

In his blog on the 2nd Village Telco Workshop, Steve Song mentioned that the Village Telco entrepreneur is likely to want tools for mass configuration of the network, e.g. putting all the nodes on a single subnet, re-numbering the phone numbers, etc. The whole mass-reconfiguration thing made me think of my music collection and the elegant simplicity of tools like EasyTag which allow you to batch edit an individual attribute of a single mp3 file or a thousand. Having an EasyTag-like editor to edit the attributes of the Mesh Potatoes on your network would make network configuration a breeze.

This page will reflect the state of the ongoing design and development of this bulk configuration tool (the "Potato Mesher" for want of a better name). I welcome all interested parties to email me and/or update this wiki page with suggestions, questions, answers, and any other helpful information.

= Preliminary Fact finding = These are [mailto:lew.pitcher@digitalfreehold.ca my] preliminary notes on such a tool, along with a series of questions that I think are relevant to the design and implementation of such a tool. The notes (captured below) are in the form of an email that I sent to Steve Song (CC'ed to David Rowe), and their responses.

The Initial Email (Fri Aug 28 17:02:40 2009)
In the absence of any real information regarding the hows and whys of the proposed "Bulk Configuration" tool, I've done some thinking, and have a broad outline of how I see such a tool being organized.

Before I get to the outline, I'd like to ask if you have anything more specific about what you expect of the tool, other than what was posted in  Steve Song's blog post on the 2nd Village Telco get-together. What I'm looking for is guidance, in the form of
 * Use case(s) or other narrative on how the Village Telco enterpreneur would use the tool,
 * Your preferred technologies, platforms, and protocols (i.e. "Linux, GTK2 using SNMP over Wifi to acquire and update configs")
 * What elements are we trying to configure? Which ones are "bulk configurable"?
 * Anything else you can think of that would make the job of design and development easier.

Now, for the outline. I've attached a preliminary mind-map of the direction I've been looking in. The tool would consist of three facilities:
 * 1) a Configuration Management facility that gathers existing MP configuration information, presents it for editing, accepts and validates any changes (including bulk changes) to the MP configurations, and packages up the changes for implementation in the target MPs
 * 2) a Configuration Distribution facility, that distributes and implements configuration changes to a list of affected MPs. This would include scheduling the implementation of changes, distribution of those changes, tracking the distribution, rollback/backout of failed changes, and reporting of failed or delayed changes
 * 3) a number of "backend" interfaces which would drive knowledge of MP changes out to the other Village Telco applications where applicable.

Configuration Management
It isn't clear to me which MPs are candidates for Bulk Configuration management, and how their configurations would be determined and altered. I can see a couple of Bulk Configuration scenarios, each with their own special requirements.


 * a) Bulk configuration of Undistributed MPs: This is where the VT has a number of "stock" MPs in his possession, and wants to configure them prior to placement in the field. For these MPs, we don't have to worry about changing a phone number on a customer (so we can depricate "Phone Book" updates), but we probably want to keep an internal record of the settings and values (in some sort of "Inventory") for later reference. Because the MPs are in the VTs physical possession, access to the device can go through the wired ethernet interface, or through Wifi.


 * b) Bulk configuration of Distributed MPs:This is where the VT has already distributed MPs to customers or locations, and wants to reconfigure them. In the case of an MP already in customer hands, a change (to the Phone Number, for instance) may impact both "Billing" (if the customer's bill is somehow keyed by or indicates the phone number) and the "Phone Book". Additionally, an affected customer should be informed of the change to his phone number. Since the MPs are already in the field, data gathering and update will either be by Wifi, or by wired connect with a site visit. For wifi, if the MP is "out of the mesh", then a site visit may be necessary anyway, for both configuration gathering and for distribution of the new configuration.


 * How do you envision the bulk configuration tool being used? Before, or after distribution? Wired or wireless? On-mesh or off-mesh?
 * Do you have any preferences how the tool handles exceptions to your use-case scenario?

In all cases, we need to gather configuration information from each target MP. I presume that some programmatic facility will collect this information, through one or more queries to each MP. There seems to be a number of good candidates for a query language and protocol; SNMP seems built for this sort of thing, but I can also see an HTTP[S] request/response to the Afrimesh software doing the same thing. Each MP should provide all values for it's own configuration, along with a unique identifier, to distinguish individual MPs for reporting and downstream updating.


 * Do you have preferences as to the protocol? The query language?

Once gathered, the configuration values are presented to the VT operator, who then works through them, making both bulk and individual changes where necessary. The VT operator can distinguish individual MPs through a unique identification value, and filter the list of MPs to remove devices that s/he does not want to change. Bulk changes are reflected in the individual settings for each MP selected, and VT operator can select individual MPs for further changes. Once happy with the changes, the VT operator commits the changes to the tool.


 * Do you have a list of the configuration fields that you want to be able to bulk change? Prefered default values?
 * Do you have a list of unique identifier fields that we can key to? Which are used by the back-end apps (like A2Billing)?

Configuration Distribution
The distribution of changed configuration values back to the affected MPs does not have to occur while the VT operator works with the interactive component. Indeed, some changes must wait on the MP to achieve a specific state (i.e. IP address or Phone Number changes may require the MP to achieve an "idle" state, neither making nor receiving phone calls, nor relaying for other MPs). Some changes may require large data transfers (changes to Asterisk dialplans or sound clips). Target MPs may be "offline", and may need manual intervention, or special scheduling before they are ready to receive changes. For all these reasons, the Distribution component needs to work asynchronously from the Configuration Management facility, and asynchronously with each target MP. It needs to be able to schedule the distribution of changes to specific MPs, determine if the MP executed the change properly, reschedule the change if needed, back out the change if needed, and report any problems to the VT operator for manual resolution if necessary.


 * Do you have any preferred method of packaging the changes for distribution? (ipkg? tar.gz?)
 * Do you have any preference as to how changes are delivered to the MPs? Wifi or wired? What protocol (SNMP? HTTP? something else?)
 * What constitutes a configuration distribution error?
 * What sort of reporting is necessary?

Backend Interfaces
I can see the potential for several back-end interfaces:

An "Inventory" interface, to record the current state of each MP. Very useful for undistributed MPs, and handy for quick access to information about MPs in the field.

A "Phone Book" interface, to report/record customer-facing changes, like Phone Number. Especially necessary when MPs are already "in the field", as customers won't be able to call an MP that they don't have the correct phone number for.

A "Gateway" interface, in the case of a VT connected to the world via PSTN or a VoIP provider. The Gateway would likely need dial-plan updates, if nothing else.

A "Billing System" interface, in the case of MPs in personal possession. Bills may include both an incoming and an outgoing phone # for reporting purposes, or may report the parties by name. If phone numbers are used as keys, then changed phone numbers will affect billing reporting. Additionally, if billing is performed based on phone number components (local calls are $, calls to next village (as indicated by phone #) are $$), then the billing system needs to know about phone number changes.


 * What back ends need information?
 * What sort of information?

Finally,

 * Who are my contacts? Who do I ask questions of?
 * Who are the contacts for the other sub-projects (like A2Billing)?

Any information you can give me would be a big help in getting this project going.

Thanks

David Rowe's response
Thank you very much for your careful thought in this area. The questions you ask are good ones. IIRC we don't have concise answers to all of them but by asking these questions you are moving the project in the right direction to get them defined and addressed by a software solution. Well done :-)

My recollection at the workshop was that we simply decided that a bulk configuration tool would be a good thing - the technical details, use cases and interfaces have not been defined past what you read on the blog post.

Steve has been on leave for the last few weeks but will probably be back in the first week of September. Lets get his input on some of your questions below when he returns.

Thanks again for your great work in this area.

Steve Song's response (Tue Sep 8 15:59:33 2009)
Sorry for the slow reply. I have indeed been on leave and less connected-while-on-leave than usual. Let me echo David's thanks for the very thoughtful laying out of the bulk configuration issues. I am very excited that you are interested in developing this.

David's right. We haven't gotten very far on bulk configuration beyond confirming the need for it. I'll try and answer your questions based on my own nascent thoughts on the issue:

Use case(s) or other narrative on how the Village Telco enterpreneur would use the tool
As you correctly identify, I think the two principal use cases are 1) initial distribution i.e. customising a stock MP for deployment on your network and 2) ongoing configuration e.g. changing the name of the upstream asterisk service, the IP subnetwork, etc, etc as the VT entrepreneur's network evolves.

Your preferred technologies, platforms, and protocols
This is up for debate. Elektra mentioned using a secure key on each MP to allow remote updating via an SSH/SCP script. Obviously the scope for screwing up the network is quite large with a bulk update so it would be very nice to have some sort of smarts that would warn you before you do something that would cut you off from the network or cut some piece of the network off from you. Advantage of a script would be that it would be very easy to call from a web interface.

On the other hand, SNMP may be more elegant. Here my opinion/expertise ends and you may have to consult the community for a more informed opinion.

What elements are we trying to configure? Which ones are "bulk configurable"?
Phone numbers, IP addresses, subnets, gateways, etc.

Having something that detects anomalies would be good. e.g. a single MP with the wrong subnet, gateway, OpenWRT version, etc.

Also an ability to serialise both IP addresses and phone numbers might be a good thing.

Anything else you can think of that would make the job of design and development easier.
Everybody works differently but my inclination on this would be to start small, develop a simple (extensible) tool that can make mass configuration changes, release it into the community and use the feedback to develop it further.

I wouldn't hesitate to share what you've developed so far with the Village Telco community either via the mailing list or by putting it on the VT wiki.

Don't hesitate to ask if there is something I can do to help you move forward on this.

= Preliminary directions = While I wait for feedback from the community, I have proceeded to lay out the design of the bulk configuration tool. My initial assumptions (at the moment) are:
 * 1) The tool will be used for the initial preparation of "factory-fresh" undistributed MPs only.
 * 2) The tool will favour existing open source software over custom code
 * 3) The tool will favour existing public protocols and interfaces over custom-designed protocols and interfaces
 * 4) The tool will run in a Linux environment
 * 5) The tool will be developed in a language, and with support toolkits and libraries that are common to Linux systems
 * 6) The tool will consist of independent components for each of data acquisition, data manipulation, and data distribution

As a preliminary direction, I am looking into using SNMP to collect the configuration information from each MP, and am now (Sept 2009) researching lightweight SNMP agents that could run on the Mesh Potato.