Thursday, October 02, 2008

iPhone Programming Tips: building Unix software

The NDA that was part of the agreement between Apple and registered iPhone developers was lifted yesterday. Not an hour had gone by before other developers started to share tips, code or even full frameworks. Like them, we'd also like to contribute some of the learnings we've found during our work with Apple's SDK.

We've decided, therefore, to write a second piece in our internationally acclaimed iPhone Programming Tips series. Our first article, devoted to image orientation techniques, was written during our jailbreak apprenticeship, before the NDA even existed - how's that for anticipation?

This time we'll explore a very different area: how to build a Unix source package in such a way it can be used as part of your iPhone project.

The Problem



So you've found this wonderful open source Unix package that perfectly handles one of the areas you need in your application. Being a publicly scrutinized, well-maintained library, you decide you would waste your time if you tried to do something similar, so you try to use it in your project and concentrate on your own features. Or maybe you simply want to test how this thing would work in the iPhone. Whatever your reasons are, you download the distribution and read the INSTALL file. This package is so well designed it compiles under a zillion Unix variants, including many evolutionary dead ends. It includes a fancy "configure" script that guesses everything it needs to know to configure itself properly. Apple's Developer Tools include a full gcc/make toolchain, so you perform the sacred ritual "./configure; make" and presto, you obtain an Intel dynamic library ready for use!

Wait a minute, that's not what you need. You want to cross-compile for ARM so that it runs on the device, of course. And you need it to run in the simulator, too.

Being a resourceful, tough programmer, you drop the source code on your Xcode project, alongside the rest of your code. You build your project and get like 148 errors and 57 warnings. Naturally, there are lots of definitions that ./configure should have defined, and the source defaults must be those of a PDP-11, or something. You try to tweak the settings manually for 10 minutes before you give up.

Your next step is to tell the configure script to cross-compile for a different platform. But you don't know exactly how to tell it your platform is like Mac OS X, only running on an arm chipset, what's the problem about that. After reading the scripts and template files and googling your way around, you use something like ./configure --host=arm-apple-darwin. This doesn't work, either, because configure insists on using your standard Mac OS X system libraries and headers instead of the ARM ones for the iPhone. You then try to tell configure to use the gcc compiler in your /Developer distribution, and this still doesn't work. Oh, well. You know what you need to do - investigate compilation options and library locations - but it's painful.

Wouldn't it be nice to have some notes on how to set up your environment to compile an Unix package from the command-line, but using Xcode's development libraries for the iPhone?

The Solution



If you have read the introduction above, you will have noticed that it is just a feeble attempt to disguise the fact that this article is only going to tell you how to configure your environment variables to compile a Unix library using Xcode's toolchain from the command line. That's it. It is admittedly not a very glamourous or innovative task; however, we still had to devote a few lenghty hours to set everything up properly, following the embarrassing steps outlined above, one after the other.

So, without further circumlocutions, let us show the damned variables. I'll dissect one of the build scripts I have actually used, accompanying the definitions with some comments or caveats.

The following two definitions point to the root of the command line developer tools and iPhone SDK. You may need to modify them to update the location in your own system, or if you are using a newer version of the SDK:


export DEVROOT=/Developer/Platforms/iPhoneOS.platform/Developer
export SDKROOT=$DEVROOT/SDKs/iPhoneOS2.0.sdk



Next, let's save the current build environment - we'll use it later to build the i386 version of the library which will run in the simulator.


# Save relevant environment
U_CC=$CC
U_CFLAGS=$CFLAGS
U_LD=$LD
U_LDFLAGS=$LDFLAGS
U_CPP=$CPP
U_CPPFLAGS=$CPPFLAGS



We'll now define the values we need to use to target the ARM architecture. Compilation flags in my case look something like this:


export CPPFLAGS="-I$SDKROOT/usr/lib/gcc/arm-apple-darwin9/4.0.1/include/ -I$SDKROOT/usr/include/"
export CFLAGS="$CPPFLAGS -arch armv6 -pipe -no-cpp-precomp -isysroot $SDKROOT"
export CPP="/usr/bin/cpp $CPPFLAGS"



Linking flags are a bit more tricky. You need to ensure the output of the compilation process is a static library, and not a dynamic library or an executable file. Dynamic libraries can in fact be produced, but the iPhone sandbox will sadly refuse to load them at runtime - only dynamic libraries and frameworks in predefined system locations can be used. There's a mention about this limitation somewhere in the Development Agreement.

Even though you won't be using dynamic libraries in your project, some packages are configured in such a way that it is easier to let them compile the dylib then ignore it, rather than trying to convince them not to create the dynamic code. If you encounter such a case, you should configure your linker in a way similar to this:


# dynamic library location generated by the Unix package
LIBPATH=src/.libs/<libname>.dylib
LIBNAME=`basename $LIBPATH`

export LDFLAGS="-L$SDKROOT/usr/lib/ -Wl,-dylib_install_name,@executable_path/$LIBNAME"



This will create a valid dylib file that you will be able to use within Xcode; however, it won't run in the iPhone as described above.

Therefore, you will actually be interested in using the static library version, so we'll store its location:


# static library that will be generated
LIBPATH_static=src/.libs/<libname>.a
LIBNAME_static=`basename $LIBPATH_static`



Now we are ready to run the configure script and build the libraries.


./configure CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin
make



Depending on the package you are trying to compile, you might need to supply additional arguments to the configure script. Some packages, for example, will accept arguments indicating whether a static library or a dynamic one should be built. It is also frequent to disable features or modules you know you won't use in your project. You need to refer to your package documentation for fine tuning details.

After make finishes (hopefully without errors), we'll move away the generated libraries to a safe location:


mkdir -p lnsout
cp $LIBPATH_static lnsout/$LIBNAME_static.arm



We repeat now the same steps, but targetting the i386 architecture. This will allow us to build libraries compatible with our iPhone simulator environment.


make distclean

# Use default environment
export CC=$U_CC
export CFLAGS=$U_CFLAGS
export LD=$U_LD
export LDFLAGS=$U_LDFLAGS
export CPP=$U_CPP
export CPPFLAGS=$U_CPPFLAGS

# Overwrite LDFLAGS
# Dynamic linking, relative to executable_path
# Use otool -D to check the install name
export LDFLAGS="-Wl,-dylib_install_name,@executable_path/$LIBNAME"

# ToDo - error checking
./configure
make

# Save generated binaries
cp $LIBPATH_static lnsout/$LIBNAME_static.i386



After we have produced the i386 and arm versions of our library, we will create a fat Universal Binary enclosure containing both of them:


# Create fat lib
$DEVROOT/usr/bin/lipo -arch arm lnsout/$LIBNAME_static.arm -arch i386 lnsout/$LIBNAME_static.i386 -create -output lnsout/$LIBNAME_static



Finally, you can add the generated library and any necessary development header files to your project and build it. If everything went well, your application will be linked with the library and will run correctly in both the simulator and the device.

As explained above, this is mostly basic Unix tinkering, but it took us a while to get the right configuration. Maybe we were just rusty, so if you know a better way to achieve this, please do let us know!

36 comments:

Anonymous said...

Thank you! That will certainly come in useful and save alot of dicking around in the future. :)

Unknown said...

Hi,

Wow, I just spent the entire evening trying to figure this stuff out and here it is!

I did manage to get some arm files out of my attempt to compile lighttpd, but I'm not sure if they actually work just yet. I will re-do my attempt using your techniques.

You wouldn't happen to have already compiled lighttpd with mod_webdav support would you have? I'd like to add a small webserver to my iPhone application so users can make backups of my iPhone app's data.

Thanks,

Brendan

Unknown said...

Hmm.. When I try to go through the steps of compiling lighttpd for the iPhone using your steps, I get this error:

cp $LIBPATH_static lnsout/$LIBNAME_static.arm
cp: src/.libs/.a: No such file or directory


It seems that it's not compiling a static library for some reason. Here's the configure command I'm using:

./configure CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin --disable-ipv6 --enable-static --disable-shared

Any ideas? I think you guys build Sketch and have integrated a web server into it. Did you use lighttpd? If so, would you care to share your steps for building it? Or post the binary somewhere? Does it work both on the device and the simulator? How did you get the webserver up and running from within your iPhone app?

Thanks!

Brendan

Pedro said...

Hi Brendan,

You need to define LIBPATH_static to point to the actual location of the library your package generates. That location is totally up to the maintainers of the software, so you need to check the specificities of lighttpd. You could try to build it for Intel first using a standard run, then note the location of the file.

I have corrected a formatting mistake in the code box where LIBPATH_static is defined above - it should be clearer now that the path is a placeholder that needs to be replaced with the correct value for the package.

Sketches integrates a web server, but we are using libmicrohttpd instead of lighttpd, so I'm afraid we can't help with that.

Good luck!

Unknown said...

Ok, thanks for your response. I hadn't heard about libmicrohttpd before. However, I need WebDAV support, so I'll keep working on lighttpd. Unless you know of another embedded webserver that includes WebDAV support?

Thanks!

Brendan

Unknown said...

Ok, thanks for your response. I hadn't heard about libmicrohttpd before. However, I need WebDAV support, so I'll keep working on lighttpd. Unless you know of another embedded webserver that includes WebDAV support?

Thanks!

Brendan

Unknown said...

Oops. Sorry. Didn't mean to double-post. I had troubles with the captcha and I guess it took my response twice.

monsta said...
This comment has been removed by the author.
monsta said...
This comment has been removed by the author.
Anonymous said...

I got it to compile with the help of this site and Alice at gotcandy.com/alice

If your interested the link is http://monstamacs.blogspot.com/

thanks for your useful blogging !

Ryan H said...

How would you set up the environment to find other static libraries during the build process so that they can be integrated into the build?

For example, building openssl with zlib. If I have a static zlib build, how would I set things up so that during the build of openssl, it finds the zlib library?

Anonymous said...

Hi,

I was trying this with imagemagick without success:
calling configure with:
./configure CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin --disable-largefile --with-quantum-depth=8 --without-magick-plus-plus --without-perl

The configure runs without an error, but the make stops with:
magick/random.c:103:25: error: crt_externs.h: No such file or directory
make[1]: *** [magick/magick_libMagickCore_la-random.lo] Error 1
make: *** [all] Error 2


but it is existing: /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator2.1.sdk/usr/include/crt_externs.h

What am I doing wrong?

regards,

Ralf

Ryan H said...

Try adding this to your CFLAGS:
-I$SDKROOT/usr/include

either by exporting it in the shell:

EXPORT CFLAGS = "$CFLAGS -I$SDKROOT/usr/include"

or adding to your config

./configure CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 CFLAGS="$CFLAGS -I$SDKROOT/usr/include" LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin --disable-largefile --with-quantum-depth=8 --without-magick-plus-plus --without-perl

Anonymous said...

Thanks, but this was already in my settings - I made a script with the information here:
export DEVROOT=/Developer/Platforms/iPhoneOS.platform/Developer
export SDKROOT=$DEVROOT/SDKs/iPhoneOS2.1.sdk

# Save relevant environment
U_CC=$CC
U_CFLAGS=$CFLAGS
U_LD=$LD
U_LDFLAGS=$LDFLAGS
U_CPP=$CPP
U_CPPFLAGS=$CPPFLAGS


export CPPFLAGS="-I$SDKROOT/usr/lib/gcc/arm-apple-darwin9/4.0.1/include/ -I$SDKROOT/usr/include/"
export CFLAGS="$CPPFLAGS -arch armv6 -pipe -no-cpp-precomp -isysroot $SDKROOT"
export CPP="/usr/bin/cpp $CPPFLAGS"

# dynamic library location generated by the Unix package
LIBPATH=libs/imagemagick.dylib
LIBNAME=`basename $LIBPATH`

export LDFLAGS="-L$SDKROOT/usr/lib/ -Wl,-dylib_install_name,@executable_path/$LIBNAME"

# static library that will be generated
LIBPATH_static=src/.libs/imagemagick.a
LIBNAME_static=`basename $LIBPATH_static`

./configure --build=i686-pc-cygwin CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin --disable-largefile --with-quantum-depth=8 --without-magick-plus-plus --without-perl

make

It still ends with:
magick/random.c:103:25: error: crt_externs.h: No such file or directory
make[1]: *** [magick/magick_libMagickCore_la-random.lo] Error 1
make: *** [all] Error 2

Anonymous said...

ok ;-) I see ... this just don't exist for the iPhone.
It exists for MacOD and the iPhone Simulator, but not the iPhone.
I think I have to provide an implementation myself.

rko_cri said...

Hi, I was following your tips building imagemagick - the problem with crt_externs.h was that is was not existing for the iphone sdk. So after solving this I got after compiling:
./libtool: eval: line 961: syntax error near unexpected token `|'
I figured out that may be the entry

# Take the output of nm and produce a listing of raw symbols and C names.
global_symbol_pipe=""

I am not shure what is expected here so I set it to global_symbol_pipe="cat"

Then I end up with:
/bin/sh ./libtool --silent --tag=CC --mode=link /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-gcc-4.0.1 -std=gnu99 -I/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.1.sdk/usr/lib/gcc/arm-apple-darwin9/4.0.1/include/ -I/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.1.sdk/usr/include/ -arch armv6 -pipe -no-cpp-precomp -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.1.sdk -Wall -W -D_THREAD_SAFE -no-undefined -export-symbols-regex ".*" -module -avoid-version -L/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.1.sdk/usr/lib/ -o coders/art.la -rpath /usr/local/lib/ImageMagick-6.4.8/modules-Q8/coders coders/coders_art_la-art.lo magick/libMagickCore.la
link: illegal option -- d



Do you have any hint for me?

Anonymous said...

I can't get the very last part to work.

lipo complains about the unknown architecture type 'arm'

-arch arm

Any suggestions anyone?

Ryan H said...

Are you using armv6 or just arm? you should specify armv6.

Also, check to make sure you're setting your sysroot correctly (-isysroot ...)

Anonymous said...

Plz dOnt do boring unix stuff on this excellent multimedia phone.

Unknown said...

Hi,

I'm trying to do what you're talking about for the openLDAP libraries and getting the following errors...

Ade$ CC=$DEVROOT/usr/bin/arm-apple-darwin9-gcc-4.0.1 CFLAGS=-isysroot/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.2.1.sdk LD=$DEVROOT/usr/bin/ld ./configure --host=arm-apple-darwinconfigure: WARNING: If you wanted to set the --build type, don't use --host.
If a cross compiler is detected then cross compile mode will be used.
Configuring OpenLDAP 2.4.16-Release ...
checking build system type... i686-apple-darwin9.6.0
checking host system type... arm-apple-darwin
checking target system type... arm-apple-darwin
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking for arm-apple-darwin-strip... no
checking for strip... strip
checking configure arguments... done
checking for ar... ar
checking for style of include used by make... GNU
checking for arm-apple-darwin-gcc... /usr/bin/arm-apple-darwin9-gcc-4.0.1
checking for C compiler default output file name...
configure: error: C compiler cannot create executables
See `config.log' for more details.


I adapted it slightly but still get the following errors..

Ade$ AGS=-L/usr/local/BerkeleyDB.4.7/lib CPPFLAGS=-I/usr/local/BerkeleyDB.4.7/include CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-gcc-4.0.1 CFLAGS=-isysroot/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.2.1.sdk ./configure --host=arm-apple-darwin
configure: WARNING: If you wanted to set the --build type, don't use --host.
If a cross compiler is detected then cross compile mode will be used.
Configuring OpenLDAP 2.4.16-Release ...
checking build system type... i686-apple-darwin9.6.0
checking host system type... arm-apple-darwin
checking target system type... arm-apple-darwin
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk...

...checking how to run the C preprocessor... /lib/cpp
configure: error: C preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.

I'm at a loss here. Any help that sheds light on this situation would be greatly appreciated!


Cheers,
Ade

Unknown said...

Hi,

There's a discussion going on in the Apple Discussion Lists that I think you would be able to help with.

It is to do with linking a static library and stopping Xcode from looking for the dynamic ones...

http://discussions.apple.com/thread.jspa?messageID=9356160#9356160


Cheers,
Ade

Ryan H said...

I haven't tried building open LDAP, but sometimes it's helpful to look at the port files that MacPorts use to help you figure out how to set up and configure your environment. If you have MacPorts installed, I suggest you take a look at: /opt/local/var/macports/sources/rsync.macports.org/release/ports/databases/openldap/Portfile

You should try to get it to build for the default architecture first, then modify for arm.

Matt said...

I'm trying to compile the tesseract ocr engine for the iPhone, and I think I've almost got it.

I get the error cp $LIBPATH_static lnsout/$LIBNAME_static.armcp: src/.libs/tesseract.a: No such file or directory
which I assume means I haven't specified the location of the library properly.

When trying to find this location, I found about 10 possible files, with names like libtesseract_main.a and libtesseract_ccstruct.a. Is there one in particular that I specify, or do I need to tell it about all of them?

If I install for the Mac, it places all 10 of the .a files in urs/local/lib.

Pedro said...

Hi Matt,

I'm not familiar with the Tesseract engine (although it does sound cool!), but you probably need to either create a library for each of the ones supplied in the package, or maybe combine them all in a single file.

I would try to create separate files first. In order to do so, you need to define them in your Makefile, and adapt the source and target destinations to their appropriate names. So, instead of using

# static library that will be generated
LIBPATH_static=src/.libs/<libname>.a

You would need to use several similar definitions, each pointing to one of the output files in the package. This approach might not be the optimum one, but I believe it should work.

Good luck!

Cloud said...

You can find a detailed post with a script to have ImageMagick build as a static library with support to jpeg and png over at: http://www.cloudgoessocial.net/2009/06/09/imagemagick-on-iphone-with-jpeg-png/

Best regards!

Tom Bottiglieri said...

Have you come across a package that was dependent on Glib (and in turn, gettext and libiconv)?

I am having a bear of a time trying to cross compile all of these into static libraries.

Unknown said...

I post here iphone cross compilation the list of library I successfully cross-compiled for iphone. Thank you for your help.

Prabhakar said...

Anyone had luck compiling with net-snmp?If so i would like to have instructions.

Thanks in Advance

Unknown said...

It's a nice information for me to done my work.Thanks for it.also chk

unlock iphone

mobile software said...

I've tried to compile it for 10.5 and with sdk3.0 and I get such message, how to enable arm ?

/usr/bin/lipo: unknown architecture specification flag: arm in specifying input file -arch arm lnsout/libtesseract_full.a.arm
/usr/bin/lipo: known architecture flags are: any little big ppc64 x86_64 ppc970-64 ppc i386 m68k hppa sparc m88k i860 veo ppc601 ppc603 ppc603e ppc603ev ppc604 ppc604e ppc750 ppc7400 ppc7450 ppc970 i486 i486SX pentium i586 pentpro i686 pentIIm3 pentIIm5 pentium4 m68030 m68040 hppa7100LC veo1 veo2 veo3 veo4

Manpreet Singh said...

Thanks for putting together a nice set of instructions for the iphone.

I noticed a problem that someone posted about in the comments with the error message:
./libtool: eval: line 961: syntax error near unexpected token `|'

I encountered the same was able to fix this by defining AS and NM from the cross compile chain. Basically, to the instructions posted here, also add:

export AS=$DEVROOT/usr/bin/as
export NM=$DEVROOT/usr/bin/nm

anev said...
This comment has been removed by the author.
anev said...

'm trying to compile libpcap 1.1.1 for use in another project but it fails saying it cannot find headers. This is the configure line i'm giving it;

bash-3.2# ./configure CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-gcc-4.0.1 LD=$DEVROOT/usr/bin/ld --host=arm-apple-darwin

configure: WARNING: If you wanted to set the --build type, don't use --host.
If a cross compiler is detected then cross compile mode will be used.
checking build system type... i386-apple-darwin10.2.0
checking host system type... arm-apple-darwin
checking target system type... arm-apple-darwin
checking for arm-apple-darwin-gcc... /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-gcc-4.0.1
..
configure: error: pcap type not determined when cross-compiling; use --with-pcap=...

Looking at the config.log file I get the following:

bash-3.2# grep "error:" config.log
conftest.c:10: error: ac_nonexistent.h: No such file or directory
conftest.c:53:25: error: sys/bitypes.h: No such file or directory
conftest.c:20: error: sys/bitypes.h: No such file or directory
conftest.c:29:23: error: net/pfvar.h: No such file or directory
conftest.c:28:30: error: netinet/if_ether.h: No such file or directory
conftest.c:32:30: error: netinet/if_ether.h: No such file or directory
conftest.c:63:27: error: netinet/ether.h: No such file or directory
conftest.c:30: error: netinet/ether.h: No such file or directory
conftest.c:38:30: error: netinet/if_ether.h: No such file or directory
conftest.c:47: error: invalid application of 'sizeof' to incomplete type 'ac__type_new_'
configure:6902: error: pcap type not determined when cross-compiling; use --with-pcap=...

Kimberly said...

It seems to me that apple are excellent programs, for example, in occasion I worked at a casino called free casino in which you used the mac operating system, and it was excellent ...

Unknown said...

hi,
i just wanted to say how much i enjoy reading your blog. in a world full of spin, it's nice to get some fact-based analysis.
keep up the good work.
iphone

Thomas said...

How would you set up the environment to find other static libraries during the build process so that they can be integrated into the build?
Penny Auction