Using files from arago-oe-dev in arago overlay

While working with the Arago overlay I've been faced with the problem of needing "files" from the OpenEmbedded snapshot (arago-oe-dev) in my recipes that are contained in the Arago overlay. This can include .inc files or even files specified in the SRC_URI as file:// something. I've usually been able to avoid this issue in the first place by placing my recipe in the OpenEmbedded mainline (which is my preferred way to do things) or just amending a recipe in the OE mainline (i.e. adding an amend.inc file to the Arago overlay). However, recently I was confronted with a situation in which this wouldn't work.

The issue was that I had a recipe that does the following:

1. It patches Qt/embedded to use library optimized to run on the ARM NEON.
2. The library is NOT available for mass download (at least not at this time). Instead it is only available inside of the TI network.

The above conditions meant that I could not really put this recipe in the OE mainline because unless you were building from inside of the TI network it would fail to build. I also couldn't just amend the OE recipe in Arago because of the same reason, I would prevent the Arago mainline from being able to build Qt/e for anyone outside of TI. So what was my solution?

I decided to create a new recipe call qt4-embedded-blitrix_4.6.2.bb in the Arago overlay that would PROVIDES "qt4-embedded". Then, when building inside of TI I could add a conf/arago-bom.conf file (which is already included in local.conf) to my directory and specify PREFERRED_PROVIDER_qt4-embedded = "qt4-embedded-blitrix". In this way regular Qt builds would work but I could easily specify when to build the TI only version of Qt.

Here is where the file sharing comes in. Obviously I don't want to have to copy all of the files and .inc files from the OE snapshot into the Arago overlay. So I needed a way in my recipe to add the arago-oe-dev (OE snapshot) directory to my files path. Here is what I did:

1. For .inc files I just made them relative paths which then causes bitbake to search the BBFILES paths for the first instance of the .inc file. So instead of saying "require qt4-embedded.inc" I instead said "require recipes/qt4/qt4-embedded.inc"
2. For files in the SRC_URI I needed to add to my FILESPATHBASE variable the oe snapshot directory (arago-oe-dev). This would allow my recipe to find files in the arago-oe-dev directory as well as in the arago overlay directory. To do this I added FILESPATHBASE .= ":${OEBASE}/arago-oe-dev/recipes/qt4". Notice that I appended this path as I would want to get files from the Arago overlay first.
3. Lastly, because my recipe had a different PN value (because of the name of the recipe file) I needed to add the original qt4-embedded directories to the list of places to search for these files. To do that I did FILESPATHPKG .= "qt4-embedded-${PV}:qt4-embedded". NOTE: depending on what files you are trying to pull you may need to prepend these values to make sure you get the right files. Or you could define the whole variable setting instead of appending or prepending. i.e. qt4-embedded:${PF}:${P}:${PN}:${BP}:${BPN}:qt4-embedded-${PV}:qt4-embedded:files:.:qt-${PV}
   

So basically the first several lines of my recipe looked like:

# Add the base qt4-embedded directories to the list of
# directories to be searched for files.
FILESPATHPKG .= ":qt4-embedded-${PV}:qt4-embedded"

# Add the arago-oe-dev directory to the base directories
# to search for files. This keeps us from having to
# copy files from the OE snapshot to the Arago overlay.
FILESPATHBASE .= ":${OEBASE}/arago-oe-dev/recipes/qt4"

# Using relative path here causes bitbake to search in
# BBPATH for the first instance of qt4-embedded.inc rather
# than just within the current directory.
require recipes/qt4/qt4-embedded.inc


Making the above modifications kept me from having to copy all the .inc files and the files from the SRC_URI into the Arago overlay.

NOTE: This recipe is not yet finished and I still have issues to resolve. This blog is just to serve as an example of how to use files from the OE mainline in the overlay without copying them.

distribute_license.bbclass

This blog post is going to be about the distribute_license.bbclass file that was added to openembedded mainline recently. It was added as commit id 5e75fa8c1b28055b610eb47ba5d3368a16810aa4 on July 30th 2010.

The reason I created this class was that while distributing images and software packages as part of an SDK I needed to also distribute a manifest of what software components were distributed and their licensing. However, some packages like liboil had "unknown" licensing defined in the ipk control file. This is because they don't have a standard license like GPL.

Instead packages like liboil have a COPYING or LICENSE or some other type of file in their soruces that indicates their licensing terms. The purpose of the distribute_license class is to take these files from the sources and copy them into the deploy direcgtory. Then the meta sdk package can include them for distribution. Thus packages like liboil can have their license set to "liboil" and users can look at the liboil directory in the deploy/licenses directory to find the license files for liboil.

By default this class will look only in the top-level source directory of a package for files matching the pattern COPYING* or LICENSE*. However, each package has the ability to define its own LICENSE_FILES pattern as well as LICENSE_SEARCH_DEPTH. This can be done using overrides such as:

LICENSE_FILES_liboil = "COPYING"
LICENSE_SEARCH_DEPTH_liboil = "2"

The reason for this is that packages may use a non-standard name for their license files. Thus it is useful to be able to change the search string. They may also not include the license file in the top-level directory, or sub-directories could have different licenses. This is why we allow varying search depth.

When the search depth is expanded past the top-level directory any matching licenses in a sub-directory will be placed in the ${DEPLOY_DIR}/licenses/${PN}/ relative path. That way it is clear what directory each license file corresponds to.

Downgrading packages in Ubuntu

So I needed to downgrade the version of my gzip package in Ubuntu 10.04 because it wouldn't work with my openembedded build (I was getting errors unpacking the expat tarball). What I ended up doing was adding the "hardy" packages to my list of sources through the software sources gui (deb http://security.ubuntu.com/ubuntu hardy-security main). Then I followed the instructions here to find the older version of gzip and downgrade it. Now my build works. Basically what I did was:

1. System->Administration->Software Sources
2. Go to Other software and click Add
3. Add the line above (deb http://security.ubuntu.com/ubuntu hardy-security main)
4. Click close and allow it to rebuild the database.
5. run apt-cache showpkg gzip
6. You'll see the versions. You want the 1.3.12-3.2ubuntu0.1 version
7. run sudo apt-get install gzip=1.3.12-3.2ubuntu0.1
8. Select yes when asked if you want to downgrade.

A quick note on conf files

So I wanted to have my own configuration file in OE that I could use to specify things like which packages to make source ipks for or what my PREFERRED_PROVIDERs were for things like the virtual/kernel package.

This let me to the question of how can I easily add a new configuration file? I figured that some files must just be picked up automatically but was it really just as simple as adding another line to something like my local.conf? The simple answer is YES.

You can just add a line like:

include /.conf

The only thing to be aware of is that you can either "include" a configuration file, which means that it will be used if it exists but if it doesn't exist that's OK. Or you can "require" a configuration file, which as the name implies means it has to be present or you will get an error.

As for adding a configuration file "automatically" I'm not sure how to do that and I have it on good advice that you don't want to do that. I guess I can understand that because it is nice to know explicitly how something got added and forget all the automagical stuff.

What does virtual/xyz point to?

Often I see things in recipes where they rely on things like "virtual/kernel". But how do I know what bitbake recipe virtual/kernel points to?

Usually you can find this information in the machine configuration file. This isn't always guaranteed, it could be stashed in some other configuration file like the distro configuration file (i.e. Angstrom defines the PREFERRED_PROVIDER_virtual/update-alternatives)

In the case of the dm3730-am3715-evm machine type its configuration file specifies:

PREFERRED_PROVIDER_virtual/kernel = "linux-omap-psp"

This tells me that if I were to "bitbake virtual/kernel" it would look for the latest linux-omap-psp recipe and build that with the exception of whether or not that recipe set DEFAULT_PREFERENCE to -1. If so then it would look for the next version of linux-omap-psp.

So in general the virtual/xyz values are set in the configuration files. You could of course always just "bitbake virtual/xyz" and see what it builds, but if you have a slow system like mine sometime you want to know without waiting on a build :)

What is SOC_FAMILY?

So I was playing around with adding new tasks for the dm3730-am3715-evm machine type (I'll be submitting patches for these soon) and I was getting really annoyed because I kept picking up packages that I didn't want. For example I had the following section in one of my tasks:

MULTIMEDIA = ""

MULTIMEDIA_omap3 = " \
ti-dmai-apps \
ti-codec-engine-examples \
task-arago-gst \
gstreamer-ti \
"

RDEPENDS_${PN} = "\
${MULTIMEDIA} \
"


I was expecting that since I didn't have a MULTIMEDIA_dm3730-am3715-evm variable defined that I would pick up the default MULTIMEDIA line and there would be no RDEPENDS setting. Instead I kept getting the omap3 settings which was trying to build packages I didn't want.

So I dug into the conf/machine/dm3730-am3715-evm.conf file and noticed that it included the omap3.inc file which sets SOC_FAMILY to omap3. This gave me my insight into what was happening. The omap3 lines were getting picked up because the dm3730-am3715-evm section was empty. Basically the order was MACHINE -> SOC_FAMILY -> DEFAULT.

I'm sure this is somewhat simplified, but it gets the point across. Basically as I see it the SOC_FAMILY is a good way to specify variables for an entire family of devices without having to say XYZ_dev1, XYZ_dev2, etc and set them all the same.

writing my source ipk class for OE

So as part of my work for building TI SDKs I needed a good way to distribute the sources for a package wrapped in an ipk package. This is because our newer installers actually use opkg to install the various components of the SDK onto the user's host system. Going forward this will allow users to select the components that get installed in their SDK with finer granularity.

But, if we are installing these packages we really need a good way to install the sources that correspond with them. This is where the sourceipk.bbclass comes into play. I should take a moment here and clarify that this source ipk is NOT intended as a method for building the binaries of a package like you might be used to when thinking of things like a source rpm. It is only for distributing the sources of a package.

So some goals:

• I don't want to have to modify a bunch of base recipes to add this feature. This causes divergence from the mainline.
  
• I want to distribute the patched sources as well as the patches and recipe
  
• I want to have control per package over whether a source ipk is built, what architecture it is built for, and where it will install on the host system.

Now what did I do?

I created a sourceipk.bbclass file in the classes directory of OE and added a "INHERIT += sourceipk" line to my configuration file (could be local.conf, angstrom.conf, whatever). This class adds a step between the do_patch and do_configure steps that will make a copy of the sources (which have now been patched) and generate an ipk in the deploy directory with the form (package name)-src_(version)_(architecture).ipk.

But just adding the INHERIT line is not enough. By default this class won't actually do anything unless you specify a non-zero value for CREATE_SRCIPK for the package you want (see the comments in the sourceipk.bbclass file for some more details). Basically you have to say per package whether you actually want a source ipk generated or not. I did this because there are more packages where I don't want/need the sources installed on the host system than packages that I do want the sources for. For example I may not need to have the dropbear sources on my customers system but I definitely need to give them the Linux kernel sources so they can rebuild their kernel.

So how do I specify which packages get source ipks built?

There are two ways to do this. For testing or if you are writing a new recipe and want people to get a source ipk by default you can add the following lines to your bitbake recipe:

CREATE_SRCIPK = "1"
SRCIPK_INSTALL_DIR = "where/do/you/want/it/installed"
SRCIPK_PACKAGE_ARCH = "architecture"
inherit sourceipk

The SRCIPK_INSTALL_DIR and SRCIPK_PACKAGE_ARCH lines are actually optional because the will default to "/usr/src/${PN}-src" for the SRCIPK_INSTALL_PREFIX and "all" for the SRCIPK_PACKAGE_ARCH if you don't specify them.

But remember that one of my goals was to NOT have to change a bunch of recipes to make source ipks. This is where the second method comes in. In the second method I can add the inherit of sourceipk into a configuration file and then define the values for each package in that file. For example I can modify my angstrom-2008.1.conf file and add the following lines:

INHERIT += "sourceipk"
CREATE_SRCIPK_pn-helloworld = "1"
SRCIPK_INSTALL_DIR_pn-helloworld = "demos/"

CREATE_SRCIPK_pn-linux-omap-psp = "1"
SRCIPK_INSTALL_DIR_pn-linux-omap-psp = "psp/kernel"
SRCIPK_PACKAGE_ARCH_pn-linux-omap-psp = "armv7a"

The above lines mean that I will generate a source ipk for the helloworld and linux-omap-psp packages when they are built. In the case of helloworld I want it installed on my target system in a directory called demos whereas I want the Linux kernel installed in a "psp/kernel" directory. I also specified a different architecture for the Linux kernel package (armv7a). This was just to show that it can be done. Usually you want to leave this as "all" so that the sources can be installed on any system.

Another quick note about the install directory. When installing the ipks for the TI SDK we specify a "-o" option to the opkg program. That way if I say that helloworld should install in the "demos" directory I can use a -o value of "/home/user/SDK" and it will actually install helloworld in the "/home/user/SDK/demos" directory. This is important to remember so that you don't install into the root of your development host.

Some interesting issues I faced:

• I originally had the sourceipk.bbclass adding a new package to the PACKAGES list so that it would just use the normal packaging system. That all seemed to be working well until I hit packages like the Linux kernel which decides to just reset/overwrite the PACKAGES variable. This meant that while my code would get executed and the sources made ready to go the default packaging system wouldn't make the package. So instead I decided to just call ipkg-build myself to create the package explicitly. I found the packaged-staging.bbclass sources helpful as well as the IPKG Howto site.
• When the patches were applied to the sources a "patches" directory was created in the sources directory containing all the patches, or so I thought. What it turned out to be was that they were symbolic links to the patches that were applied. So when I was doing a copy I was getting the symbolic link. When I would later install the source ipk I would get a bunch of messages about being unable to create the symbolic links because the files weren't there. What I needed to do was add the -L flag to my cp command so that I would copy the actual file pointed to by the symlink and not the symlink itself.

I still have an issue with trying to use the ipkg-build script. If I use that script I have to add a dependency on ipkg-utils-native to the sourceipk.bbclass. This produces a circular dependency because ipkg-utils-native will inherit sourceipk which in turn depends on ipkg-utils-native. For the mean time I have taken the pieces I needed of the ipkg-build script and put them into the sourceipk.bbclass directly. See the TODO comment in the sourceipk file for more details and if you know how to fix this issue please feel free to share.

OK, so that was a long blog post but I wanted to note what I had been working on, why I made it, and how it works. I also tried to document this in the comments and header of the sourceipk.bbclass code. I hope this was helpful.