Jump to: navigation, search

How To Set Up a SECN Build Environment

Revision as of 21:12, 4 April 2016 by Terry Gillett (talk | contribs) (Setting the OpenWrt Code Revision: Added trunk repo note)

Author: Terry Gillett

Introduction

This HowTo document is a simplified description of the process for setting up an OpenWrt build environment for producing firmware for VillageTelco SECN devices.

There are two parts to this document. The first part describes how to set up a build environment using pre-prepared Setup scripts, and the second part describes how to set up the build environment manually step by step. The end result of the two different processes is essentially the same.

The script based setup is designed reproduce the published SECN firmware, using the same versions of all the software repositories used to build the original firmware. It is based on the newer Git based source repositories.

Setting up the build environment manually will allow you to further customise the build process, and potentially choose different versions of the component software repositories. The manual process described here is based on the older Subversion based software repositories.

References: OpenWrt Wiki » Documentation » HOWTOs » OpenWrt Buildroot – Installation

            http://wiki.openwrt.org/doc/howto/build
            http://wiki.openwrt.org/doc/howto/buildroot.exigence


Scripted Setup of a Build Environment

At the time of writing, Setup scripts using the Git based repositories are available for SECN 3 firmware, based on OpenWrt BB 14.07, and for SECN 4 firmware, based on OpenWrt CC 15.05

Once the Setup script has been run to create the build environment, then the Build scripts are used to build firmware for specific devices.

The VT SECN source code is held in the GitHub repository at:

   https:// github.com/villagetelco/vt-firmware

SECN 3 source code is contained in the "secn_3.0" branch, and SECN 4 source code is held in the "secn_4.0" branch. You will need to check out the appropriate branch as below to get the required Setup script. The Build scripts will automatically check out the appropriate branch when they are run.

For the purposes of this document, we assume that you will have a directory "~/Git" that will contain the local Git repositories, and a directory " ~/openwrt" that will contain the build environments. The Build scripts assume this directory structure by default, so if you change it you will have to amend the Build scripts accordingly.

You need to have installed the necessary Git utilities as follows:

   $ sudo   apt-get   update
   $ sudo   apt-get   install    -y   git-core   subversion   build-essential   gawk

Create a local copy of the vt-firmware repository using commands as follows:

   $ mkdir ~/Git
   $ cd ~/Git
   $ git clone  https://github.com/villagetelco/vt-firmware

This will create a directory called "vt-firmware". To set up a build environment for SECN 3 for example, you need to checkout the appropriate branch of the repo as follows:

   $ cd   ~/Git/vt-firmware
   $ git   checkout   secn_3.0

If you look in the vt-firmware directory you will see a directory called "Setup-scripts" which contains the file "Setup-14.07.sh"

Copy this file into the "~/openwrt" directory, make it executable, then run it as below. The script will download several hundred MB of data so make sure you have a good Internet connection set up before you start.

   $ cd ~/openwrt
   $ chmod   +x    Setup-14.07.sh
   $ ./Setup-14.07.sh   ~/openwrt/Build-14.07

This will create a new directory called "Build-14.07" and download the required software sources. Various progress messages will be seen on-screen, and the details saved in log files for debugging if necessary.

The Setup script defines particular revisions of the OpenWrt source code and of the various source code repositories (including packages, telephony, routing, fxs) so that the resulting firmware builds will have the same software configuration as the published SECN firmware.

You can change the defined repository revisions by editing the "feeds.conf.default" file and running the command "$ ./scripts/feeds update -a"

However, be aware that changing the repository revisions may result in incompatibilities between packages and/or the underlying OpenWrt version, so proceed with caution.


Using the Build Scripts

Look in the "~/Git/vt-firmware" directory and you will see a folder called "Build-scripts". There are build scripts for MP02, TP-Link and Ubiquity devices.

Copy the appropriate device build script (eg Build-MP02.sh) into the new "Build-14.07" directory.

Edit the script and look at the beginning of the script to find the version strings that will be used to name the firmware files to be produced, and edit if/as required.

When the firmware is built, it will have a unique build date/time stamp included internally in the command line splash page, and in the directory name, so that individual firmware can always be distinguished, even if the version strings are the same. Be aware that if you build the exactly same firmware again, it will have a different md5sum due to the internal date stamp.

Look at end of the Build script to see the list of devices that will be built. Comment out the lines for devices that you do not require.

Then change to the build directory and run the build script as follows:

   $ cd ~/openwrt/Build-14.07
   $ ./Build-MP02.sh

The initial build will take quite quite a long time and will download a lot of data, so make sure you have a good, reliable Internet connection. The directory may grow to about 5GB in size so make sure there is adequate disc space available.

Subsequent builds will be considerably faster as the various component packages will have already been built.

The build script has optional lines for the main "make" command to take advantage of multi-core CPUs on your PC (eg make -j5 for a four core machine). It is advisable to run it with the default ( make -j1 ) for the initial build. While this is slower, it ensures that the build has the best chance of completing successfully.

For subsequent builds you can comment out this line and un-comment the multi-core line. Be aware that using the multi-core command may produce random build errors.

This "make" command also has a the flag "V=s" set by default. This provides a verbose output which is also saved in a log file so that you can look through it in the event that the build fails for some reason. Again, this is useful to have in place for the initial build.

One reason that builds can fail is that a particular set of software is temporarily not available for download from the Internet. This will be flagged at the end of the build log so you can see what has happened.

When the build completes successfully, the firmware and md5sums files will be saved in the "Builds" directory, and in a subdirectory according to the device type, and named for the particular device, build type and date. Each time you run a build script a new directory will be created to hold the firmware files produced by that build run.


Customising the Setup Scripts

The Setup scripts by default use specific versions of the OpenWrt codebase and packeges. Because the version of the OpenWrt codebase and the packages are locked to specific versions, anyone setting up a build environment with the script will be able to generate the same firmware.

If different versions of the OpenWrt codebase or the packages are required, then these can be selected as required by editing the script asdescribed below.

Setting the OpenWrt Code Revision

The OpenWrt codebase has branches corresponding to major releases such as Barrier Breaker (14.07) and Chaos Calmer (15.05)

The details of these branches are available here:

   https://dev.openwrt.org/browser/branches

Within a branch, there will be updates from time to time.

The revision number shown against the branch on the above web page represents the latest update to the branch (eg chaos calmer r48666).

The Setup scripts allow you to either take the latest updated version of the branch, or to specify a particular update version by specifying the SHA value for the revision in Git.

To see all the revisions of a particular branch, you can check out the Git repository for the branch and browse through it with a suitable Git browser such as Gitg.

To get a copy of the repositories for BB and CC releases, use the following commands:

  $ cd ~/Git
  $ git   clone   git://git.openwrt.org/14.07/openwrt.git   ./openwrt-14.07
  $ git   clone   git://git.openwrt.org/15.05/openwrt.git   ./openwrt-15.05

To get a copy of the repository for trunk, use the following command:

  $ git   clone   git://git.openwrt.org/openwrt.git   ./openwrt-trunk


This will create directories ~/Git/openwrt-14.07 , ~/Git/openwrt-15.05 and ~/Git/openwrt-trunk containing the repositories which can be browsed with a tool such as Gitg.

To update the repository at any time use the commands:

  $ cd ~/Git/openwrt-14.07       or       $ cd ~/Git/openwrt-15.05        or       $ cd ~/Git/openwrt-trunk
  $ git  pull  origin


The Setup script contains a code block at the top of the script similar to the following:

  # Set default version parameters
 REVISION="15.05"
 SETSHA="TRUE" 
 SHA="48e7befb"    # 15.05 branch as at 5 Aug 2015

If SETSHA is set to FALSE, then the latest revision of the branch will be used.

If SETSHA is set to TRUE, then the revision corresponding to the specified SHA value will be used.

By default, the Setup scripts have SETSHA set to TRUE and so will use a revision with an explicitly specified SHA value, generally corresponding to the revision used to build the current stable version of the SECN code.

Setting the version of Packages

Similarly, by default, the script will lock the feeds for various packages to specific versions in the "feeds.conf.default" file.

You can edit this file to change the version of one or more package feeds if required. The original file is saved as "feed.conf.default.bak"

To update the feeds after making a change, use the following commands:

 $  cd ~/openwrt/Build-14.07
 $  ./scripts/feeds update -a


Setting up a Build Environment Manually

This section of the document assumes that you are setting up a build environment on Ubuntu 14.04 LTS and using the "Barrier Breaker" revision of OpenWrt.

References: OpenWrt Wiki » Documentation » HOWTOs » OpenWrt Buildroot – Installation

            http://wiki.openwrt.org/doc/howto/build
            http://wiki.openwrt.org/doc/howto/buildroot.exigence

Prerequisites

   - A PC with Ubuntu operating system installed. 
     The process below has been tested on Ubuntu 14.04 LTS
   - 500 MB of hard disk space for the source files to be downloaded.
   - 5 GB of available hard disk space to build (i.e. cross-compile) OpenWrt.

Notes

   - Do everything as non-root user! 
     Use your normal login account to issue commands, using sudo where indicated.
   - Issue all OpenWrt Buildroot commands in the <buildsystem root> directory
     e.g. ~/openwrt/attitude/
   - Do not try to build in a directory that has spaces in its full path.

Install Git and Subversion

Git and Subversion (svn) allow you to conveniently download the OpenWrt source code and build tools to assist with the compilation process.

Install Git, Subversion and the build-essential tools as follows:

    sudo apt-get  install  git-core  subversion  build-essential  gawk

For information about Git see git [1]

For information about the subversion tool see svn [2] and subversion [3] documentation.

For information about make and the build tools see make [4] and build-essential [5].

Download the OpenWrt sources

For general information about OpenWrt code versions see:

 http://wiki.openwrt.org/about/history


To see recent updates to various branches of the source code, check these sites:

   http://dev.openwrt.org/browser/branches

Click on a branch name to expand and see details of changes, for example:

   http://dev.openwrt.org/browser/branches/attitude_adjustment
   http://dev.openwrt.org/browser/branches/barrier_breaker


Note: SECN firmware on the VT Downloads site is generally built .with the latest revision for the particular branch (eg AA, BB) as shown in the above pages at the time of the build.


The required source code can be checked out of the Subversion repository and downloaded as follows:

    cd
    mkdir ~/openwrt
    cd openwrt

For the original Attitude Adjustment 12.09 stable release (r36420):
    svn checkout --revision=36420 svn://svn.openwrt.org/openwrt/branches/attitude_adjustment/   my-dir

For an updated Attitude Adjustment 12.09 release (for example r42647):
    svn checkout --revision=r42647 svn://svn.openwrt.org/openwrt/branches/attitude_adjustment/  my-dir

For an updated Barrier Breaker release (r43321):
    svn checkout --revision=r43321 svn://svn.openwrt.org/openwrt/branches/barrier_breaker/

Or for a particular trunk (development) revision e.g. rev 36420
    svn checkout --revision=36420 svn://svn.openwrt.org/openwrt/trunk    my-dir

This will create a directory 'openwrt' in your home directory, then check out the required revision into a new sub-directory:

    ~/openwrt/my-dir

which will contain the OpenWrt source code, and will include the OpenWrt Buildroot system.


Note: To view the contents of the SVN repository use the command:

    svn ls svn://svn.openwrt.org/openwrt/branches/
    svn info svn://svn.openwrt.org/openwrt/branches/

    svn ls svn://svn.openwrt.org/openwrt/branches/attitude_adjustment
    svn info svn://svn.openwrt.org/openwrt/branches/attitude_adjustment

    svn ls svn://svn.openwrt.org/openwrt/branches/barrier_breaker
    svn info svn://svn.openwrt.org/openwrt/branches/barrier_breaker


Note: OpenWrt also provide pre-built firmware for all targets. To see the firmware available for download, browse to:

To see all available downloads, including trunk snapshots:
    http://downloads.openwrt.org/

For the original Attitude Adjustment 12.09 stable release:
    http://downloads.openwrt.org/attitude_adjustment/12.09

For the original Barrier Breaker 14.07 stable release:
    http://downloads.openwrt.org/barrier_breaker/14.07/

Set up Package Feeds

Backup the file 'feeds.conf.default' in the build directory.

    cp ./feeds.conf.default  ./feeds.conf.default.orig

Open the feeds.conf.default file for editing and edit the line that checks out the package feed for the specified OpenWrt revision.

This is usually the first line in the file, and may already be set to the correct value, depending on how you checked out the code.

Edit the line as follows:

 
For Attitude Adjustment 12.09 stable release:   
    src-svn packages svn:[email protected]

For the updated Attitude Adjustment 12.09 release:   
    src-svn packages svn:[email protected]

Or for a particular trunk revision e.g. rev 36420
    src-svn packages svn:[email protected]


From Attitude Adjustment 12.09 RC2 onwards, batman-adv is no longer included by default in the OpenWrt build, so the feed has to be added if you are building a version of OpenWrt later than AA RC2.

Similarly for Telephony / Asterisk in Barrier Breaker.

Add lines for the feed for batman-adv, telephony and for ALFRED:

For Attitude Adjustment 12.09 Stable release:
    src-git routing  git://github.com/openwrt-routing/packages.git;for-12.09.x

For Barrier Breaker RC release:
    src-git telephony http://git.openwrt.org/feed/telephony.git
    src-git routing https://github.com/openwrt-routing/packages.git;for-14.07

For trunk release:
    src-git routing  git://github.com/openwrt-routing/packages.git

And for all:
    src-git alfred   git://git.open-mesh.org/openwrt-feed-alfred.git

You may also comment out any unwanted feeds that are active e.g. x-wrt, luci

Download Feeds

Download feeds using the 'feeds' script:

    ./scripts/feeds update -a


Install additional packages

Install additional packages including batman-adv and openssh-sftp-server:

    ./scripts/feeds install kmod-batman-adv  
    ./scripts/feeds install openssh-sftp-server
    ./scripts/feeds install usb-modeswitch
    ./scripts/feeds install usb-modeswitch-data 
    ./scripts/feeds install kmod-usb-serial 
    ./scripts/feeds install kmod-usb-serial-option 
    ./scripts/feeds install kmod-usb-serial-wwan
    ./scripts/feeds install haserl 
    ./scripts/feeds install xinetd
    ./scripts/feeds install muninlite
...

Alternatively install all packages with:

    ./scripts/feeds install -a  

Check for missing Ubuntu system packages

Check for missing Ubuntu packages on the system on which you want to build OpenWrt:

    make prereq

This will list missing Ubuntu operating system packages needed to successfully build OpenWrt using buildroot. Install any missing packages using the normal package management commands (e.g. apt-get install <package_name>), for example on Ubuntu 14.04:

sudo apt-get update
sudo apt-get install zlib1g-dev
sudo apt-get install libncurses5-dev  
sudo apt-get install gawk
sudo apt-get install git-core 
sudo apt-get install gettext
sudo apt-get install libssl-dev

Set up the Device Configuration

Run the 'menuconfig' graphical tool to set up the details of the required build:

    make menuconfig

A guide to commands that can be used in the tool are shown at the top of the screen.

For basic navigation around the page, use the arrow keys, up and down to highlight items, left and right for Select and Exit to move between pages.

To select an item for inclusion in the build, enter "Y", or "N" to deselect.


Select Device

Select the required device family and type in the menu.

For MP-01, Ubiquity Pico, Nano etc based on AR231x, or MP-01 select:

   Target System   'Atheros AR231x/AR5312' 
   Target Profile  'Default'

For TP-Link WR703N, MR11U, WR842ND, MR3020, WDR4300 etc, select:

   Target System   'Atheros AR71xxx/AR...' 
   Subtarget       'Generic'
   Target Profile  'TP-Link TL-WR703N'      Select required device type.


For Ubiquity Pico-M etc select 'Atheros AR71xxx/AR...' select:

   Target System   'Atheros AR71xxx/AR...' 
   Subtarget       'Generic'
   Target Profile  'Ubiquity Products'


NOTE: Using the AR231x image from this build environment for MP-01 will not support the telephony subsystem.

A special build environment is required to support the build of the device drivers etc.

A separate wiki article will cover that.


Select Target Image Formats

The build process will produce a variety of firmware file formats. The most commonly used format is 'squashfs' and this is produced by default.

By default, the 'jffs' and 'tar.gz' formats will also be selected and you can de-select these options if you do not require these formats.

Select Packages

Select the required packages for the VT SECN build located in the menu as indicated:

    wireless-tools         Base System
    openssh-sftp-server    Network        > SSH          See Note 1 
    uhttpd                 Network        > Web servers
    uhttpd-mod-tls         Network        > Web servers  See Note 1 re SSL support
    xinetd                 Network
    kmod-fs-ext4           Kernel modules > File Systems
    kmod-fs-vfat           Kernel modules > File Systems
    kmod-nls-cp437         Kernel modules > Native Language Support
    kmod-nls-iso8859-1     Kernel modules > Native Language Support
    kmod-batman-adv        Kernel modules > Network Support
    kmod-usb-ohci          Kernel modules > USB Support
    kmod-usb-serial        Kernel modules > USB Support
    kmod-usb-storage       Kernel modules > USB Support
    muninlite              Administration
    comgt                  Utilities
    haserl                 Utilities
    iwinfo                 Utilities
    pxg5                   Utilities
    usb-modeswitch         Utilities
...

Note 1: The use of sftp-server requires selection of 'openssl' support for uhttpd-tls, 
but uses an additional 100kB of flash memory. 
On devices with 4MB Flash, do not select openssh-sftp-server and leave the selection of SSL 
support for uhttpd-tls as the default 'cyassl' to ensure firmware size does not exceed limit.


When exiting the 'menuconfig' graphical tool, select 'Save Config' to save the configuration you have created.

The configuration will be stored in the file:

 '~/openwrt/attitude/.config'
OR
 '~/openwrt/rev-34386/.config'

This file can be copied and stored for later reuse.

NOTE: If you copy a saved '.config' file back into the build directory later, it is necessary to either run the 'menuconfig' tool again or run the 'defconfig' command, which is equivalent to opening the 'menuconfig' tool and closing it again without making any changes.

    make defconfig


NOTE: The package 'uhttpd-mod-tls' defaults to using the 'cyassl' library. If a make process is run with this selection, and subsequently the selection is changed to 'openssl', it will be necessary to do a 'make clean' of the uhttpd package to avoid a build time error.

    make package/uhttpd/clean

Also, in this scenario, the 'cyassl' library will still be loaded, taking up quite a bit of flash memory space which is critical on small devices. You have to look in the 'Libraries' section of the 'menuconfig' menu and manually de-select the cyassl library.

Add Custom Files

By default, the 'make' process will build a firmware image with the default OpenWrt configuration for the selected device. To add your own files to customise the firmware, create a './files' directory and build the directory structure you want to overlay on the base files in the build e.g

  
    ./files/www/cgi-bin/my-cgi-script.sh
    ./files/etc/config/network
    ./files/etc/config/batman-adv
    ./files/etc/init.d/my-init-file.sh


These files will be installed on the device, replacing any default files of the same name. In this way the configuration and operation of the device can be customised.

For VT firmware, the 'files' directory and '.config' file required to build the standard firmware images are generally published in the VillageTelco GitHub repository.

Proceed with the Build

Run 'make' to build the firmware images. The process will take longer to run the first time as the sources are compiled. It may take 20 minutes or more on a typical Laptop PC. Subsequent builds will be faster as only changes will be re-compiled.

    make


Collecting the output firmware files

The output files will be placed in the directory

   '~/openwrt/attitude/bin/ar71xx'  

OR

   '~/openwrt/rev-34386/bin/ar71/atheros

depending on the device family selected.



Notes

Operating System Prequisites

Additional pre-reqs may be required depending on the Ubuntu version you are using. Some examples follow:

Ubuntu: $ sudo apt-get install build-essential subversion libncurses5-dev zlib1g-dev gawk flex

Ubuntu 9.10 $ sudo apt-get install gcc-multilib bison autoconf screen gcc g++ binutils patch bzip2 flex make gettext unzip libc6 git-core

Ubuntu 11.10 $ sudo apt-get install build-essential subversion git-core libncurses5-dev zlib1g-dev gawk flex quilt

Ubuntu 64bit $ sudo apt-get install build-essential subversion libncurses5-dev zlib1g-dev gawk gcc-multilib flex git-core gettext

Using Release Sources (stable)

To check out the source code that the stable 'backfire' or 'attitude_adjustment' release is built from (plus the latest backported fixes from trunk):

    svn checkout svn://svn.openwrt.org/openwrt/branches/backfire
OR
    svn checkout svn://svn.openwrt.org/openwrt/branches/attitude_adjustment


To check out a 'tagged' version of the 'backfire' sources (ie no backports):

    svn checkout svn://svn.openwrt.org/openwrt/tags/backfire_10.03
OR
    svn checkout svn://svn.openwrt.org/openwrt/tags/attitude_adjustment_12.09/

To see what 'tagged' versions are available:

   svn ls svn://svn.openwrt.org/openwrt/tags/

Using Development Sources (bleeding edge)

The development branch (trunk) contains everything from documentation to experimental patches.

    svn checkout svn://svn.openwrt.org/openwrt/trunk/


Downloading Sources

Reference: https://dev.openwrt.org/wiki/GetSource


Troubleshooting

Beware of unusual environment variables such as GREP_OPTIONS which should not have '-initial-tab' or other options affecting its output

SED should not be set. If it is, run `unset SED` before compiling.