Wednesday, June 30, 2010

writing my source ipk class for OE

So as part of my work for building TI SDKs I needed a good way to distribute the sources for a package wrapped in an ipk package. This is because our newer installers actually use opkg to install the various components of the SDK onto the user's host system. Going forward this will allow users to select the components that get installed in their SDK with finer granularity.

But, if we are installing these packages we really need a good way to install the sources that correspond with them. This is where the sourceipk.bbclass comes into play. I should take a moment here and clarify that this source ipk is NOT intended as a method for building the binaries of a package like you might be used to when thinking of things like a source rpm. It is only for distributing the sources of a package.

So some goals:
  • I don't want to have to modify a bunch of base recipes to add this feature. This causes divergence from the mainline.
  • I want to distribute the patched sources as well as the patches and recipe
  • I want to have control per package over whether a source ipk is built, what architecture it is built for, and where it will install on the host system.
Now what did I do?

I created a sourceipk.bbclass file in the classes directory of OE and added a "INHERIT += sourceipk" line to my configuration file (could be local.conf, angstrom.conf, whatever). This class adds a step between the do_patch and do_configure steps that will make a copy of the sources (which have now been patched) and generate an ipk in the deploy directory with the form (package name)-src_(version)_(architecture).ipk.

But just adding the INHERIT line is not enough. By default this class won't actually do anything unless you specify a non-zero value for CREATE_SRCIPK for the package you want (see the comments in the sourceipk.bbclass file for some more details). Basically you have to say per package whether you actually want a source ipk generated or not. I did this because there are more packages where I don't want/need the sources installed on the host system than packages that I do want the sources for. For example I may not need to have the dropbear sources on my customers system but I definitely need to give them the Linux kernel sources so they can rebuild their kernel.

So how do I specify which packages get source ipks built?

There are two ways to do this. For testing or if you are writing a new recipe and want people to get a source ipk by default you can add the following lines to your bitbake recipe:

CREATE_SRCIPK = "1"
SRCIPK_INSTALL_DIR = "where/do/you/want/it/installed"
SRCIPK_PACKAGE_ARCH = "architecture"
inherit sourceipk

The SRCIPK_INSTALL_DIR and SRCIPK_PACKAGE_ARCH lines are actually optional because the will default to "/usr/src/${PN}-src" for the SRCIPK_INSTALL_PREFIX and "all" for the SRCIPK_PACKAGE_ARCH if you don't specify them.

But remember that one of my goals was to NOT have to change a bunch of recipes to make source ipks. This is where the second method comes in. In the second method I can add the inherit of sourceipk into a configuration file and then define the values for each package in that file. For example I can modify my angstrom-2008.1.conf file and add the following lines:

INHERIT += "sourceipk"
CREATE_SRCIPK_pn-helloworld = "1"
SRCIPK_INSTALL_DIR_pn-helloworld = "demos/"

CREATE_SRCIPK_pn-linux-omap-psp = "1"
SRCIPK_INSTALL_DIR_pn-linux-omap-psp = "psp/kernel"
SRCIPK_PACKAGE_ARCH_pn-linux-omap-psp = "armv7a"

The above lines mean that I will generate a source ipk for the helloworld and linux-omap-psp packages when they are built. In the case of helloworld I want it installed on my target system in a directory called demos whereas I want the Linux kernel installed in a "psp/kernel" directory. I also specified a different architecture for the Linux kernel package (armv7a). This was just to show that it can be done. Usually you want to leave this as "all" so that the sources can be installed on any system.

Another quick note about the install directory. When installing the ipks for the TI SDK we specify a "-o" option to the opkg program. That way if I say that helloworld should install in the "demos" directory I can use a -o value of "/home/user/SDK" and it will actually install helloworld in the "/home/user/SDK/demos" directory. This is important to remember so that you don't install into the root of your development host.

Some interesting issues I faced:

  • I originally had the sourceipk.bbclass adding a new package to the PACKAGES list so that it would just use the normal packaging system. That all seemed to be working well until I hit packages like the Linux kernel which decides to just reset/overwrite the PACKAGES variable. This meant that while my code would get executed and the sources made ready to go the default packaging system wouldn't make the package. So instead I decided to just call ipkg-build myself to create the package explicitly. I found the packaged-staging.bbclass sources helpful as well as the IPKG Howto site.
  • When the patches were applied to the sources a "patches" directory was created in the sources directory containing all the patches, or so I thought. What it turned out to be was that they were symbolic links to the patches that were applied. So when I was doing a copy I was getting the symbolic link. When I would later install the source ipk I would get a bunch of messages about being unable to create the symbolic links because the files weren't there. What I needed to do was add the -L flag to my cp command so that I would copy the actual file pointed to by the symlink and not the symlink itself.
I still have an issue with trying to use the ipkg-build script. If I use that script I have to add a dependency on ipkg-utils-native to the sourceipk.bbclass. This produces a circular dependency because ipkg-utils-native will inherit sourceipk which in turn depends on ipkg-utils-native. For the mean time I have taken the pieces I needed of the ipkg-build script and put them into the sourceipk.bbclass directly. See the TODO comment in the sourceipk file for more details and if you know how to fix this issue please feel free to share.

OK, so that was a long blog post but I wanted to note what I had been working on, why I made it, and how it works. I also tried to document this in the comments and header of the sourceipk.bbclass code. I hope this was helpful.

My first post

OK, so now I have a blog going and I guess I have to do something with it. I've generally avoided blogging and twittering and things like that but I find that I should probably start writing down some of the things I am working on. I work for Texas Instruments focusing on open source platforms and methods. Currently I'm focused on things like the beagleboard and the future OSD3 device from Neuros. I'm also working a lot on how TI will build and distribute our software development kits going forward. This is going to be an OpenEmbedded based solution using an overlay called Arago. My blog is primarily going to focus on these topics.

Hopefully I'll keep up with my posting (remember I don't really like social media).