README.zxid
###########
<<author: Sampo Kellomäki (sampo@iki.fi)>>
<<cvsid: $Id: README.zxid,v 1.78 2007-08-10 19:19:10 sampo Exp $>>
<<class: article!!!ZXID 19>>

<<abstract:

ZXID.org Identity Management toolkit implements standalone SAML 2.0 and
Liberty ID-WSF 2.0 stacks and aims at implementing all popular
federation and ID web services protocols.  It is a C implementation
with minimal external dependencies - OpenSSL, CURL, and zlib -
ensuring easy deployment (no DLLhell). Due to its small footprint and
efficient and accurate schema driven implementation, it is suitable
for embedded and high volume applications. Language bindings to all
popular highlevel languages such as PHP, Perl, and Java, are provided
via SWIG.  ZXID implements, as of July 07, SP, WSC, and WSP roles. IdP
role will follow as the project evolves.

ZXID.org ist eine C-Bibliothek, die den vollständigen SAML
2.0-Stack implementiert und alle populären
Identitätsverwaltungs-Protokolle wie Liberty ID-FF 1.2,
WS-Federation, WS-Trust und ID-Webservices wie Liberty ID-WSF 1.1 und
2.0 implementieren will. Sie beruht auf Schema-basierter
Code-Erzeugung, woraus eine genaue Implementation resultiert. SWIG
wird verwendet, um Schnittstellen zu Skriptsprachen wie Perl, PHP und
Python sowie zu Java bereitzustellen. Sie kann als SP, IdP, WSC und
WSP fungieren.

A biblioteca de gestão de identidades ZXID.org é uma
implementação, em C, das normas SAML 2.0 e Liberty ID-WSF 2.0 com
dependências externas mínimas - OpenSSL, CURL, e zlib - facilitando
uma implantação fácil sem "inferno dos DLL". Sendo económica em
consumo de recursos é indicada para aplicações embutidas ou
de grande volume e performance. A biblioteca é disponibilizada para
todos os linguagens de programação de alto nível como, p.ex.,
PHP, Perl, e Java, atravez de interfáces SWIG. ZXID de hoje (Jul
07) pode funcionar nos papeis SP (Provedor de Serviços), WSC
(Cliente de Serviços Web) e WSP (Provedor de Serviços Web),
sendo o papel IdP (Provedor de Identidade) suportado
na futura evolução do projecto.

La libraria de gestión de idendidades ZXID.org es una
implementación, en C, de las normas SAML 2.0 y Liberty ID-WSF 2.0
con dependencias extranjas minimas - OpenSSL, CURL, y zlib - que
elimina el "DLL Hell" en implantación. Como ZXID es mui economica
es adepto para aplicaciones embutidos o de gran volumen y
envergadura. Los lenguas de programación de alta nivel, como Perl,
PHP, y Java, son suportados con gerador de interfaces SWIG. Hoy (Jul
07) el ZXID suporta papeles SP (proveédor de servicios), WSC
(cliente de los servicios web) y WSP (proveédos de los servicios web).
Lo papel IdP (proveédos de los identidades) sea suportado
en fase futuro de el proyecto.

ZXID.org on elektonisenhenkilöllisyyden ja -tunnisteiden hallinta
kirjasto joka tukee SAML 2.0 (sisäänkirjaantuminen) ja Liberty
ID-WSF 2.0 (henkilöllisyyteen pohjautuvat webbipalvelut)
standardeja. ZXID on riippuvainen vain OpenSSL, CURL ja zlib
kirjastoista joten se välttää "DLL helvetti"
-ongelman. Skemapohjaisena C toteutuksena se on tarkka ja
taloudellinen ja soveltuu sulautettuihin ja erittäin kovaa
suorituskykyä vaativiin sovelluksiin. Se tukee korkeantason
kieliä, kuten Perliä, PHP:tä, ja Javaa, SWIG generoiduin
rajapinnoin. ZXID tukee (Heinäkuu 07) SP (palveluntarjoaja), WSC
(webbipalvelunkutsuja), ja WSP (webbipalveluntarjoaja) rooleja.
IdP (henkilöllisyydenvarmentaja) rooli toteutetaan projektin tulevissa
vaiheissa.

>>

<<maketoc: 1>>

1 Who needs this?
=================

ZXID project has currently (Jan 2007) five outputs

libzxid:: A C library for supporting SAML 2.0, including federated
    Single Sign-On (SSO)
zxid:: A C program that implements a SAML Service Provider (SP) as a CGI script
Net::SAML:: A Perl module wrapping libzxid. Also zxid.pl, that implements SP
    in mod_perl environment, is supplied.
php_zxid:: A PHP extension that wraps libzxid. Also supplied: zxid.php
    that implements SP in mod_php environment.
libzxidjni.so:: A Java JNI extension that wraps libzxid. Also supplied:
    zxid.java that implements SP as a CGI script.

*You need this if you are*

Web Master:: You want to enable SAML based Single Sign-On (SSO) to
    your web site. In this case you would use the zxid SP CGI script
    directly, only configuring it slightly or you can go the zxid_simple()
    route. Otherwise you can hint your PHP or perl developer that this
    functionality is available and your want it.

Perl Developer:: You can use the Net::SAML module to integrate SSO
    to your application and web site. Given the direct perl support, this is
    easier than fully understanding the C interface. Both mod_perl
    and perl as CGI are supported.

PHP Developer:: You can use ~dl("php_zxid.so")~ to load the module and
    access the high level functionality, such as SAML 2.0 SSO. We
    support functionality roughly equivalent to perl Net::SAML.
    The PHP module is fully ready to use for SSO, but we expect to
    add a lot more, such as WSC, in future. Both mod_php5 and php as
    CGI are supported. php4 should also work.

Java Developer:: You can use ~System.loadLibrary("zxidjni")~ to
    pull into your Java proram the full power of the ZXID. The
    functionality supported is roughly equal to Net::SAML.

Web Developer:: You want to integrate SAML based SSO to your web site tool
    or product so that your customers can enjoy SSO enabled web sites. In
    this case you would study zxid.c for examples and use libzxid.a to
    implement the functionality in your own program.

Identity Management hacker:: You need some building blocks: you
    will study libzxid and add to it, contributing to the project.

ZXID Project has vastly more ambitious goals. See the ZXID Project chapter
later in this document.

2 Installing
============

If you want to try ZXID out immediately, we recommend compiling the
library and examples and installing one of the examples as a CGI
script in an existing web server. See later chapters for more details.

  tar xvzf zxid-0.15.tgz
  cd zxid-0.15
  # N.B.  There is no configure script. The Makefile works for all
  #       supported platforms by provision of correct TARGET option.
  # N.B2: We distribute some generated files. If they are missing, you need
  #       to regenerate them: make cleaner; make dep ENA_GEN=1
  make                   # default Linux. Do `make TARGET=sol8' for Solaris
  make dir               # Creates /var/zxid hierarchy
  
  make samlmod           # optional
  make samlmod_install   # optional: install Net::SAML perl module
  make phpzxid           # optional
  make phpzxid_install   # optional: install php_zxid.so PHP extension
  make javazxid          # optional

  cp zxid <webroot>/
  # configure your web server to recognize zxid a CGI, e.g.
  mini_httpd -p 8443 -c 'zxid*' -S -E zxid.pem

  # Edit your /etc/hosts to contain
  127.0.0.1       localhost sp1.zxidcommon.org sp1.zxidsp.org

  # Point your browser to (zxid_simple() API version)
  https://sp1.zxidsp.org:8443/zxidhlo?o=E
  https://sp1.zxidsp.org:8443/zxidhlo.pl?o=E       # Perl version
  https://sp1.zxidsp.org:8443/zxidhlo.php?o=E      # PHP version
  http://sp1.zxidsp.org:8080/zxidservlet/zxidHLO?o=E  # Java version

  # Point your browser to (full API version)
  https://sp1.zxidsp.org:8443/zxid?o=E
  https://sp1.zxidsp.org:8443/zxid.pl?o=E       # Perl version
  https://sp1.zxidsp.org:8443/zxid.php?o=E      # PHP version
  https://sp1.zxidsp.org:8443/zxid-java.sh?o=E  # Java version

  # Find an IdP to test with and configure it...

2.1 Prerequisites
-----------------

This software depends on the following packages:

1. zlib from zlib.net. Generally whatever comes with your distro is sufficient.
2. openssl-0.9.8d or later. See www.openssl.org. Generally openssl libraries
   distributed with most Linux distros are sufficient.<<footnote: It is
   possible to compile without OpenSSL, e.g. for space constrained embedded
   system, but this has serious security implications.>>
3. libcurl from http://curl.haxx.se/. I used version 7.15.5, but probably
   whatever ships with your distribution is fine. libcurl is needed
   for SOAP bindings and for fetching metadata. It needs to be compiled
   to support HTTPS.<<footnote: Compilation without libcurl is possible
   with some loss of functionality.>>
4. HTTPS capable web server. For most trivial testing CGI support is needed. We
   recommend mini_httpd(8) available from
   http://www.acme.com/software/mini_httpd/
5. Perl, PHP, and Java interfaces depend on the respective development
   tools but should not need any additional modules or tools.

Following additional packages are needed by developers who wish
to build from scratch, including the code generation (the standard
distribution includes the output of the code generation, so most
people do not need these).

A. gperf from gnu.org (only for build process when generating code)
B. swig from swig.org (only for build process and only if you
   want scripting interfaces)
C. perl from cpan.org (only for build process and only if you
   want to generate code from .sg)
D. plaindoc from http://mercnet.pt/plaindoc/pd.html (only for
   build process, for code generation from .sg, and for documentation)

Although technically not needed to build zxid, you will need an IdP
to test against. We do not, at the time, supply one, so you
will need to find a third party, perhaps a free download of one of the
commercial ones like http://symlabs.com/Products/SFIAM.html or
http://www.lasso.org/

2.2 Canned Tutorial: Running ZXID as CGI under mini_httpd
---------------------------------------------------------

While zxid will run easily under Apache httpd (see <<link:apache.html:
recipe>>), for sake of simplicity we first illustrate running it with
mini_httpd(8), a very simple SSL capable web server by Jef Poskanzer.

2.2.1 Getting and installing mini_httpd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can download the source for mini_httpd from
http://www.acme.com/software/mini_httpd/

You should already have installed OpenSSL, or quite probably OpenSSL
shipped with your distribution. If it is not located at
/usr/local/ssl, the you need to edit the mini_httpd ~Makefile~ to
indicate where it is. At any rate you need to uncomment all lines that
start by SSL_ in the ~Makefile~. Then say

  make

Now copy the mini_httpd binary somewhere in your path.

2.2.2 Running mini_httpd
~~~~~~~~~~~~~~~~~~~~~~~~

After building zxid, cd to zxid directory and run

  mini_httpd -p 8443 -c 'zxid*' -S -E zxid.pem

where

  -p 8443      specifies the port to listen to
  -c 'zxid*'   specifies that URL paths with "zxid" are CGI scripts
  -S           specifies that https is to be used
  -E zxid.pem  specifies the SSL certificate to use

See <<link:apache.html: Apache recipe>> for alternative that
avoids mini_httpd, but is more complicated otherwise.

> N.B. The zxid.pem certificate and private key combo is shipped with zxid
> for demonstration purposes. Obviously everybody who downloads zxid
> has that private key, so there is no real security what-so-ever.  For
> production use, you must generate, or acquire, your own private
> key-certificate pair (and keep the private key secret). See Certificates
> chapter for further info.

2.2.3 Accessing ZXID
~~~~~~~~~~~~~~~~~~~~

Edit your /etc/hosts file so that the definition of localhost also
includes sp1.zxidcommon.org and sp1.zxidsp.org domain names, e.g:

  127.0.0.1       localhost sp1.zxidcommon.org sp1.zxidsp.org

Point your browser to

> https://sp1.zxidsp.org:8443/zxid

or if you do not want the common domain cookie check

> https://sp1.zxidsp.org:8443/zxid?o=E

*Dynamic linking problems*

If accessing the URL (while running mini_httpd) you get
no error message and no content - everything just
mysteriously fails - you may be hitting a dynamic
linking problem. If mini_httpd(8) fails to launch CGI
script it will silently fail. This is unfortunate, but
I guess that is what the "mini" in the name implies.

To make matters even worse, mini_httpd(8), probably in the
interest of security, will ignore ~LD_LIBRARY_PATH~
variable. Apparently it has its fixed notion of the
library paths that is set at compile time.

If you suspect this problem, try following:

1. Create shell script called test.sh:

     #!/bin/sh
     echo Content-Type: text/plain
     echo
     echo Test $$
     echo lib_path is --$LD_LIBRARY_PATH--
     ./zxid -h 2>&1
     echo Exit value --$?--

2. Restart mini_httpd(8) like this

     chmod a+x test.sh
     mini_httpd -p 8443 -c test.sh -S -E zxid.pem -l mini.out

3. Access https://sp1.zxidsp.org:8443/test.sh - you
   may see something like

     Test 1655
     lib_path is --/usr/local/lib:/usr/lib--
     ./zxid: error while loading shared libraries: libcurl.so.3: cannot
         open shared object file: No such file or directory

   Now you at least see why it's failing (in this case the
   directory where libcurl was installed is not in mini_httpd's
   notion of LD_LIBRARY_PATH). If the zxid binary runs
   fine from comman line, try `ldd zxid' to see where it
   is finding its libraries.

Easiest dirty fix is to copy the missing libraries to one of the
hardwired directories of mini_httpd(8) (e.g. /usr/lib).  More
sophisticated fixes include using ldconfig(8), recompiling your
mini_httpd(8), or statically linking the offending library into zxid
binary.

2.2.4 Setting up an IdP
~~~~~~~~~~~~~~~~~~~~~~~

Currently zxid does not ship with an IdP (though the necessary
protocol encoders and decoders are latently available in libzxid,
should anyone wish to make an attempt to hack an IdP together).
For you to test zxid, you will need to acquire an IdP from
somewhere - any vendor whose product is SAML 2.0 certified
will do. One possible source is http://symlabs.com/Products/SFIAM.html
who have a free download, another is http://www.lasso.org/

If you do not want to install an IdP yourself (even for testing), find
someone who already runs one and ask if they would be willing to load
the metadata of your zxid SP. If you do this, you will need to get
externally visible domain names. This canned tutorial uses /etc/hosts
(see previous step) which is only visible on your own machine.

Once you get your IdP up and running, you need to make sure it accepts
the zxid SP in its Circle of Trust (CoT). This is done by placing
the metadata of the SP in right place in the IdP product configuration.
If your IdP supports automatic CoT management, just turn it on
and chances are you are done.<<footnote: On production IdP you should
understand the trust implications (i.e. no trust) of flipping automatic
CoT management on.>>

If not, you can obtain the zxid SP metadata (which is slightly different
for each install so you can't just copy it from existing install) from

> https://sp1.zxidsp.org:8443/zxid?o=B

This URL is the +well known location+ method metadata URL. It is also
the SP +Entity ID+ or Provider ID, should the IdP product ask for
this in its configuration. If the IdP product needs you to
supply the metadata manually as an xml file, just point your
web browser to the above URL and save to file, or use curl(1) or wget(1).

zxid SP, by default, has automatic fetching of IdP metadata enabled so
there is no manual configuration step needed, provided that the IdP
supports the well known location method. All SAML 2.0 certified IdP
implementations must support it (but you may still need to enable it
in configuration). See [SAML2Meta] section 4.1 "Publication and
Resolution via Well-Known Location", p.29, for normative description of this
method.

However, you will need the Entity ID (Provider ID) of the IdP. This is
the URL that the IdP uses for well known location method of metadata
sharing. You may need to dig the IdP documentation or GUI for a while
to find it. If you already have the IdP metadata as an xml file, open it
and look for EntityDescriptor/entityID. If you already have the
file, you can also import it manually by running the following command

  ./zxid -import file:///path/to/idp-meta.xml

But the preferred method still is: just let the automatic method
do its job.

2.2.5 Your first SSO
~~~~~~~~~~~~~~~~~~~~

1. Start at

   > https://sp1.zxidsp.org:8443/zxid

   or

   > https://sp1.zxidsp.org:8443/zxid?o=E

   If you had common domain cookie already in place, and you
   are already logged in the IdP, the SSO may happen
   automatically (go to step 3). The automatic experience
   will be typical when you use SSO regularly for more
   than one web site (i.e. several SPs).

   However, if you get a screen titled "ZXID SP SSO",
   you need to paste the IdP's Entity ID to the supplied field
   and click "Login". If zxid SP already obtained the metadata for the
   IdP, you may also see a button specific for your IdP (and in this
   case there is no need to know the Entity ID anymore or paste anything). 

2. Next step depends on the IdP product you are using. Usually
   a login screen will appear asking for user name and password.
   Supply these and login. You will need an account at the IdP.

3. For more slick IdPs, that's all you need to do and you will
   land right back at the zxid SP page titled "ZXID SP Management".

   > Congratulations, you have made your first SSO!

   However, some IdPs will pester you with additional questions
   and you will have to jump through their hoops. A typical
   question is whether you want to accept a federation. You do.

   Sometimes the federation question does not appear automatically
   and you need to figure out a way to create a federation
   in their user interface and how to get them to send you
   back to the SP. Sometimes the word used is "account linking"
   instead of federation.<<footnote: Vendor products are constantly
   improving in this area. From protocol perspective
   all the additional gyrations are unnecessary. Be sure
   to provide feedback to the vendor so that simpler, easier
   to use, products will emerge in future.>>

3 Configuring and Running
=========================

ZXID ships with working demo configuration so you can run it right
away and once you are familiar with the concepts, you can return
to this chapter.

ZXID uses a configuration file in default path<<footnote: See Simple
API for description on how to change this path at deplyment or run
time.>>

  /var/zxid/zxid.conf

for figuring out its parameters. If this file is not present, built-in
default configuration is used (see zxidconf.h).<<footnote: You can
override configuration options at run time by supplying fragments of
configuration using -O flags, but for CGI use you would have to use a
wrapper shell script to supply them. Hence, easier to just use the
config file.>> The built-in configuration will allow you to test
features of ZXID, but should not be used in production because it uses
default certificates and private keys. Obviously the demo private key
is of public knowledge since it is distributed with the ZXID package,
and as such it provides no privacy protection what-so-ever. For
production use you MUST generate your own certificate and private key.

Usually configuring a system involves following tasks

1. Configure web server (see your web server documentation)
   a. HTTPS operation and TLS certificate. In the minimum you need
      the main site, but you may want to configure the Common Domain
      Cookie virtual host as well.
   b. Arrange for ZXID to be invoked. This could mean configuring
      zxid, zxid-java.sh, or zxid.pl to be recognized as a CGI script, or
      it could mean setting up your ~mod_perl~ or ~mod_php~ system to call
      ZXID at the appropriate place.

2. Configure ZXID, including signing certificate and CoT with peer metadata
   a. generate or acquire certificate
   b. Obtain peer metadata (from their well known location) or
      enable +Instant CoT+ feature.

3. Configure CoT peers with your metadata. They can download your
   metadata from your well known location (which is the URL that
   is your entity ID). For this to happen you need to have web
   server and ZXID up and running.

3.1 Configuration Parameters
----------------------------

3.1.1 zxidroot (PATH configuration parameter)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The root directory of ZXID configuration files and directories. By default this
is /var/zxid and has following directories and files in it

  /var/zxid/
   |
   +-- zxid.conf  Main configuration file
   +-- pem/       Our certificates
   +-- cot/       Metadata of CoT partners (metadata cache)
   +-- ses/       Sessions
   `-- log/       Log files, pid files, and the like

3.1.2 pem
~~~~~~~~~

Directory that holds various certificates. The certificates
have hardwired names that are not configurable.

ca.pem:: Certification Authority certificates. These are used for
    validating any certificates received from peers (other sites
    on the CoT). The CA certificates may also be shipped to the
    peers to facilitate them validating our signatures. This is
    especially relevant if the certificate is issued by multilayer CA
    hierarchy where the peer may not have the intermediate CA certificates.
sign-nopw-cert.pem:: The signing certificate AND private key (concatenated
    in one file). The private key MUST NOT be encrypted (there will not
    be any opportunity to supply decryption password).
enc-nopw-cert.pem:: The encryption certificate AND private key (concatenated
    in one file). The private key MUST NOT be encrypted (there will not be
    any opportunity to supply decryption password). The signing certificate
    can be used as the encryption certificate. If encryption certificate
    is not specified it will default to signing certificate.

In addition to the above certificates and private keys, you will need
to configure your web server to use TLS or SSL certificates for the
main site and the Common Domain site. We suggest the following naming

ssl-nopw-cert.pem:: SSL or TLS certificate for main site. In order to
    avoid browser warnings, the CN field of this certificate should match
    the domain name of the site. The SSL certificate can be same as
    signing or encryption certificate.
cdc-nopw-cert.pem:: SSL or TLS certificate for Common Domain Cookie
    introduction site. In order to avoid browser warnings, the CN field
    of this certificate should match the domain name of the site. The SSL
    certificate can be same as signing or encryption certificate.

3.1.3 cot
~~~~~~~~~

Directory that holds metadata of the Circle of Trust (CoT)
partners. If +Instant CoT+ is enabled, this directory needs to be
writable at run time.

3.2 Configuration file format
-----------------------------

During zxid project development phase (ongoing as of Jan 2007), most
configuration related documentation will be kept as comments in
zxidconf.h, which you should see.

Configuration file is line oriented. Comments can be introduced with
cardinal (#) and empty lines are ignored. End of line comments are NOT
supported at this time.

Each configuration option is a name=value pair. The name is the same
as in zxidconf.h except that ZXID_ prefix does not appear.  Only
single line values are supported, but you can embed characters using
URI encoding, e.g. %0a for newline. In fact, the configuration lines
are treated as CGI variables, thus & can also be used as separator
instead of newline (and needs to be encoded if not intended as
separator).

When option is not specified, the default from zxidconf.h
prevails. Thus in the following the PATH specification is redundant.

To give some idea consider following /var/zxid/zxid.conf example:

  # Demo /var/zxid/zxid.conf file
  PATH=/var/zxid
  URL=https://sp.mydomain.com:8443/zxid
  NICE_NAME=My SP's human%0areadable name.
  #EOF

4 Logging and Audit
===================

N.B. zxidconf.h contains a wealth of logging related config options.

In serious use of SSO it is fundamental that the relying party, the SP,
WSC, or WSP, archives the digitally signed evidence that justifies its
actions. Generally this means that at least the SSO or credential
assertions have to be archived. Quite often, especially in the WSC
world, the entire SOAP response (which may be partially signed) needs
to be preserved as a proof of an authorized action or attested attributes.

To lesser extent, it is also important that the issuing party, the IdP
(or sometimes the DS, PS, WSC, or WSP), keeps records so that it can
confirm or refute the claims of the relying party -- in the minimum it
should be able to refute any obviously false claim and it should be
able to detect breaches of its own security arrangements,
e.g. situations where somebody is signing messages in its name
although internal audit trail demonstrates this to be impossible. The
IdP audit trail consists of preserving any (signed) request made by
anyone as well as preserving every (signed) response it makes.

Generally every assertion, request, and response will have its unique
ID that can be used as the primary key, or filename, for storing it in
a database. Unfortunately these namespaces are not disjoint (it is not
very well specified in any of the standards how they interact or how
wide their uniqueness properties are).<<footnote: Many rational
implementations use 128 bit random identifiers, which statistically
guarantees that there will not be collisions, but unfortunately we can
not rely on other parties to adopt this behaviour.>> The only safe
assumption is the pessimistic one: each type of object observes a
unique namespace only towards its issuer and type and hence we need to
map such namespaces to subdirectories.

4.1 Filesystem Layout for Logs
------------------------------

Please consider following layout of the log directory:

  /var/zxid/
   |
   +-- zxid.conf  Main configuration file
   +-- pem/       Our certificates
   +-- cot/       Metadata of CoT partners (metadata cache)
   +-- ses/       Sessions
   `-- log/       Log files, pid files, and the like
        |
        +-- issue/
        |    |
        |    +-- SHA1NAME/   Evidence given to an entity ID is kept in this directory
        |    |    |
        |    |    +-- a7n/   Assertions issued to the given 3rd party, named by AssertionID
        |    |    `-- msg/   Messages of any type issued to the given 3rd party, named by MessageID
        |   ...   
        |    `-- SHA1NAME2/  Evidence relating to another entity ID
        |
        +-- rely/
        |    |
        |    +-- SHA1NAME/   Evidence relating to given entity ID is kept in this directory
        |    |    |
        |    |    +-- a7n/   Assertions from 3rd parties, named by AssertionID
        |    |    `-- msg/   Messages of any type from 3rd parties, named by MessageID
        |   ...   
        |    `-- SHA1NAME2/  Evidence relating to another entity ID
        |
        +-- tmp/             Subdirectory used for atomic operations à la Maildir
        +-- act              Global activity log
        +-- err              Global error log
        `-- debug            Global debugging log

4.2 Log Line Format
-------------------

The log file is line oriented, one record per line irrespective of
line length, and plain text: binary data is generally omitted or
represented as (safe) base64. Fields are separated by exactly one
space character (0x20), except for the last free format field. Records
are separated by exactly one new line (0x0a) character (never by CRLF
sequence).

The log file format supports

1. Plain text logging
2. Signed plain text logging using either RSA-SHA1 or DSA-SHA1
3. Symmetrically encrypted logging using either 3DES or AES
4. Asymmetrically encrypted logging using RSA (or DSA?)
5. Signed and symmetrically encrypted logging
6. Signed and Asymmetrically encrypted logging

All activity and error log file lines have the following format (any one of the 3):

  # comment
  SE CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
  SE SSSS OURTS SRCTS IP:PORT SUCCENTID MSGID A7NID NID VVV RES OP PPP FMT

where

SE:: Log signing and encryption designator. In all cases
    the actual signing or encryption key is not identified on the log line.
    This will need to be determined out-of-band.

    PP:: PlainPlain: not signed and not encrypted
    Rx:: RSA-SHA1 signed (x = any encryption)
    Dx:: DSA-SHA1 signed
    Sx:: SHA1 check-summed, but not signed (SSSS is the checksum)
    xA:: Asymmetrically AES encrypted (x = any signing method)
    xT:: Asymmetrically 3DES encrypted
    xB:: Symmetrically AES encrypted (theoretical: how to safeguard the key?)
    xU:: Symmetrically 3DES encrypted (theoretical: how to safeguard the key?)
    xZ:: RFC1951 zipped (not really encryption)
    Xxx:: Experimental arrangements.

CCCC:: Safe base64 encoded log encryption blob. In case of encryption blob, the rest of
    the log fields will not appear. Decrypted logline will contain
    fields starting from SSSS.

SSSS:: Safe base64 encoded log line signature blob. If no signature, this is a dash ("-").

OURTS:: Our time stamp, format YYYYMMDD-HHMMSS.TTT where TTT are the milliseconds.
    The time is always in GMT (UTC, Zulutime).

SRCTS:: Source time stamp, format YYYYMMDD-HHMMSS.TTT. If TTT was not originally specified
    it is represented as "501". The time is always in GMT (UTC, Zulutime).

IP:PORT:: The IP address and the port number of the other end point (usually client,
    but could be spoofed, caveat emptor).

SUCCENTID:: The SHA1 name of the entity (succinct entity ID without the equals sign).

MSGID:: Message ID relating to the log line. Allows message to be fetched from
    the database or the file system. Any relates-to or similar ID is only
    available by fetching the original message. Dash ("-") if none.

A7NID:: Assertion ID relating to the log line. Allows assertion to be fetched from
    the database or the file system. If message benefits from multiple assertions,
    this is the one relating to the outermost one. Other A7NIDs are only
    available by fetching the original assertion. Dash ("-") if none. If the
    assertion is encrypted and can not be decrypted, then placehoder "-enca7n-"
    is used.

NID:: IdP assigned NameID relating to the message, if any. If the NameID is
    encrypted and can not be decrypted, then placeholder "-encnid-" is used.

VVV:: Signature validation codes
    O:: Capital Oh (not zero). All relevant signatures validate (generally assertion)
    A:: Unsupported or bad signature or message digest algorithm
    G:: Checksum of XML DSIG does not validate
    R:: The RSA layer of the signature does not validate
    N:: No signature detected.
    I:: Issuer metadata not found (or not in CoT, or corrupt metadata).
    V:: Assertion validity error (e.g. not in time range or wrong audience)
    Exx:: Extended signature validation code (generally failure)
    Xxx:: Experimental signature validation code (generally failure)

RES:: Result of the operation.
    K:: Operation was success
    C:: Operation failed because client did not provide valid input
    S:: Operation failed due to server side error
    P:: Operation failed due to policy or permissions issue
    T:: Temporary error, client was encouraged to retry
    D:: Redirect or recredential. Client was encouraged to retry.
    Exx:: Extended result (generally failure)
    Xxx:: Experimental result (generally failure)

OP:: The documented operation

    FEDNEW:: Federation and SSO request succeeded, new federation was created.
    FEDSSO:: SSO using federated ID was performed
    TMPSSO:: SSO using temporary NameID was performed
    SLO::    Single Logout was performed
    DEFED::  Defederation was performed

    For WSP the OP is the command verb that was exercised.

    For WSC the OP is the command verb preceded by capital C, e.g. "CQuery".

    Additional OP verbs may need to be specified for protocol substeps like
    artifact resolution (ART).

    Additional OP verbs may need to be specified for other logging operations like
    regular web access logs (HEAD, GET, POST).

PPP:: Operation dependent one most relevant parameter. Dash ("-") if none.

FMT:: Operation dependent free-form data. May contain spaces. Dash ("-") if none.

4.3 Log Signing and Encryption
------------------------------

Logs are enabled in the config file zxidconf.h (compile time) by ZXLOG
macros which provide default values for the log flags in ~struct
zxid_conf~. Each log flag is a bitmask of signing and encryption
options. Zero value means no logging. "1" can be used to enable
plain text logging.

Log signing may help you to argue that log evidence was (not) tampered
with. You can configure the signing level in the config file
zxidconf.h (compile time):

0:: no signing (Px)
2:: sha1 MD only (Sx)
4:: RSA-SHA1 (Rx)
6:: DSA-SHA1 (Dx)

For actual signing (options 2 and 3), the private key for signing must
be available in /var/zxid/pem/logsign-nopw-cert.pem. Note that this
file need not contain the actual certificate (but it may, it just will
not be used).

The weak point of log signing is that if the private key is stolen,
then someone can create falsified logs and the private key needs
to be available on the point where the logs are generated - thus
it is actually quite vulnerable.

Log encryption may help to keep the logs confidential.  You can
configure the configuration level in the config file zxidconf.h
(compile time):

0x00:: no encryption (xP)
0x10:: RFC1951 zip - safe-base64 (xZ)
0x20:: RSA-AES (xA)
0x30:: RSA-3DES (xT)
0x40:: Symmetric AES (xB)
0x50:: Symmetric 3DES (xU)

For RSA modes the public key for encryption must be available
in /var/zxid/pem/logenc-nopw-cert.pem. Note that the +private
key+ should *NOT* be kept in this file: the whole point of
public key encryption is that even if your server machine
is stolen, the bad guys can't access the logs - if the
private key was anywhere in the stolen machine, they will
find it.

For symmetric encryption the key is the SHA1 hash of file
/var/zxid/pem/logenc.key. Obviously this key must be kept secret,
but see the caveat about stolen machine in the previous paragraph.

All encryption modes, except for 0, RFC1951 zip compress the log line
before encryption and safe-base64 encode the result of the
encryption. All encryption modes, except 0 and 1, prefix the zipped log
line with 128 bit nonce before encrypting.

The algorithm is roughly

1. If encrypt, zip the raw log line
2. If sign, compute the signature (over zipped version if applicable)
3. Prepend signature blob to log line. If encrypting, the signature
   is embedded in binary form, otherwise it is embedded in safe-base64 form.
4. If encrypt, perform the encryption.
5. If encrypt, apply safe-base64.

The supplied tool zxlogview(1) allows the logs to be decrypted and the
signatures verified.

  ./zxlogview logsign-nopw-cert.pem logenc-nopw-cert.pem <some-log-lines

Note that for zxlogview(1) to work the logsign-nopw-cert.pem needs to
contain the public key (and need not contain the privatekey) which is
the opposite of the situation what zxid(1) needs to see in order to
sign. Similarly logenc-nopw-cert.pem needs to contain the private key
(and may contain the certificate, though this will not be used).

> N.B. While encrypted logs are cool, you should evaluate the
> gain against the incovenience: if you encrypt them, the lesser
> mortal sysadmins may not be able to debug  your installation
> because they do not know how to decrypt logs or you are not
> willing to trust them with the keys. For this reason, you
> can configure the encryption of error log separately.

4.4 Internal Crypto Formats
---------------------------

For RFC1951 zipped safe-base64 output the input to base64 encoding is

  LLSSSSZZZZZZZZZZZZZZ    -- RFC1951 zipped safe-base64

For encrypted modes the input to AES (or other symmetric cipher) is

  NNNNLLSSSSZZZZZZZZZZ    -- Note how nonce is prepended

The NNNN is used as initialization vector and actual encryption
encompasses LL, SSSS, and ZZZZ.

In RSA-AES the session key is encrypted using RSA and prepended
to the input for base64 encoding.

  KKEEEECCCCCCCCCCCCCC    -- RSA-AES: note prepended session key

NNNN:: 16 bytes of nonce. This is used as initialization vector
       for AES or 3DES cipher operated in CBC mode.
LL::   Bigendian integer representing signature length in bytes.
       0 means none. Negative values reserved for future use.
SSSS:: The signature in binary
ZZZZ:: RFC1951 zipped base64 of the payload
KK::   Bigendian integer representing encrypted session key
       length in bytes. Negative values are reserved for future use.
EEEE:: RSA encrypted session key in binary
CCCC:: Ciphertext from the symmetric cipher, including nonce.

In RSA operations RSA_PKCS1_OAEP_PADDING padding is used (PKCS #1 v2.0).

4.5 Logging Assertions
----------------------

Logging of assertions is controlled by configuration options
ZXLOG_ISSUE_A7N and ZXLOG_RELY_A7N. At least ZXLOG_RELY_A7N should be
turned on for ID-WSF web services to work correctly. Logging relied
assertions also allows detection of duplicate assertion IDs. Logging,
or not, of issued assertions does not have any operational effect and
is only for audit trail purposes.

Assertions are logged in directories depending on issuer's sha1 name.

  /var/zxid/log/rely/ISSUER-SHA1-NAME/a7n/A7N-ID-AS-SHA1

Sha1 names are used to avoid any attack through issuer entity ID or
the assertion ID being evilly crafted to contain shell metacharacters
or filesystem significant characters.

Assertions issued by ourselves follow similar pattern

  /var/zxid/log/issue/DEST-SHA1-NAME/a7n/A7N-ID-AS-SHA1

If the logfile starts by less-than character ("<") then it is in
plain text. Encrypted or signed formats will start in another way,
but are not specified at this time.

N.B. The relied-on assertions may be referenced from session objects
and used in construction of credentials for ID-WSF based web services
calls. Therefore the rely directory should not be cleaned too
aggressively: the assertions must remain there until the referencing
session expires.

4.6 Logging Requests and Responses
----------------------------------

Logging of requests and responses is controlled by ZXLOG_ISSUE_MSG and
ZXLOG_RELY_MSG. Logging, or not, messages has no operational effect
and is only for audit trail purposes. If logging of relied messages is
turned on, then it is possible to detect duplicate message IDs.

Request messages are logged in directories depending on issuer's sha1 name.

  /var/zxid/log/rely/ISSUER-SHA1-NAME/msg/REQ-ID-AS-SHA1

Sha1 names are used to avoid any attack through issuer entity ID or
the assertion ID being evilly crafted to contain shell metacharacters
or filesystem significant characters.

Responses issued by ourselves follow similar pattern

  /var/zxid/log/issue/DEST-SHA1-NAME/msg/RESP-ID-AS-SHA1

If the logfile starts by less-than character ("<") then it is in
plain text. Encrypted or signed formats will start in another way,
but are not specified at this time.

4.7 Session Storage and Bootstraps
----------------------------------

The ZXID session system serves three purposes:

1. Remember whether user has logged in. The session ID is carried
   either in a cookie or aspart of the URL.
2. Make it possible to perform Single Logout (SLO) and
   certain federation management tasks.
3. Remember the service end points (EPRs) that were either
   a. supplied as bootstrap attributes in the SSO assertion, or
   b. later discovered

The biggest complication is the requirement to remember the EPRs and
the solution currently used is to keep them as files in a per session
directory under the /var/zxid/ses tree.

  /var/zxid/
   |
   +-- zxid.conf  Main configuration file
   +-- pem/       Our certificates
   +-- cot/       Metadata of CoT partners (metadata cache)
   +-- ses/       Sessions
   |    |
   |    +-- SESID/         Each session has its own directory
   |         |
   |         +-- .ses      The session file
   |         +-- SVC,SHA1  Each bootstrap is kept in its own file
   |
   `-- log/       Log files, pid files, and the like

The session ID is an unguessable safe base64 encoded pseudorandom
number. Unguessability ensures that the session can only be
crated via SSO.

The service EPRs are XML documents whose name is coposed from
two components

  SVC,SHA1

SVC:: The service type URI, with file system unsafe characters
    (e.g. "/" and ",") folded to underscore ("_"). Purpose of
    the SVC is to allow quick identification, without opening,
    of the files that contain EPRs for a given service type.
    Only first 200 bytes of the service type are used.
SHA1:: safe base64 encoded SHA1 hash of the content of the EPR. The
    purpose of the SHA1 hash is to produce a unique identifier
    so that two distinct EPRs for same service will have
    different file names.

5 Compilation for Experts
=========================

  make cleaner
  make dep ENA_GEN=1
  make
  make all ENA_GEN=1

5.1 Build Process
-----------------

The build process of ZXID relies heavily on code generation techniques
that are not for the faint of heart. Some of these techniques, like
xsd2sg.pl were innovated for this project, while others like SWIG and
gperf are existing software.  Here and there some additional perl(1)
and sed(1) scripts are run to fix a thing or two.

<<dot: zxid-build: ZXID Build Process

margin=0

sg [label=".sg"];
i  [label=".i"];
phpi  [label="phpzxid.i"];
hc [label=".h and .c"];
pm [label=".pm and glue"];
php [label="php glue"];
gperf [label=".gperf"];
netsaml [label="Net::SAML"];

sg -> hc [label="xsd2sg.pl"];
sg -> gperf [label="xsd2sg.pl"];
gperf -> hc [label="gperf"];
hc -> hc [label="gen-consts-from-gperf-output.pl"];
hc -> libzxid [label="gcc"];
hc -> pm [label="swig"];
i  -> pm [label="swig"];
hc -> php [label="swig"];
phpi -> php [label="swig"];

libzxid -> zxid [label="ld"];

libzxid -> netsaml [label="ld"];
pm -> netsaml [label="perl, gcc, ld"];

libzxid -> php_zxid [label="ld"];
php -> php_zxid [label="gcc, ld"];

>>

Carefully study the Makefile, and this should all start to make sense. Please
note that there is no configuration script and GNU Auto-tools (GNU Autohell)
are not used. This is because they have been evaluated to be unsuitable
to this project (or to author's tastes). If the Makefile does not do what
you want, you should first study if you can change necessary variables using
localconf.mk, or if that does not work, then just edit the Makefile itself.
You can temporarily customize the variables in the Makefile by providing
new value on command line, e.g.

  make CC=cc   # override the default of CC=gcc

5.2 Special or embedded compile (reduced functionality)
-------------------------------------------------------

libzxid contains thousands of functions and any given application is
unlikely to use them all. Thus the easiest, safest, no loss of
functionality, way to reduce the footprint is to simply enable
compiler and linker flags that support dead function
elimination.<<footnote: Unfortunately the gnu ld does not support dead
function elimination. You should file this as a bug to them. If they
tell you to put every one of the 7000-some functions in a separate .c file,
consider the scalability implications of this. Read the comments in
pulverize.pl for a full scoop and an approach.>>

If you need to squeeze zxid into as minimal space as possible,
some functionality trade-offs are supported. I stress that you
should only attempt these trade-offs once you are familiar with
zxid and know what you are doing. The canned install instructions
and tutorial walk thrus stop working if you omit
significant functionality.

5.2.1 Compilation without OpenSSL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment out the -DUSE_OPENSSL flag from CFLAGS in Makefile and
recompile.

This will cripple zxid from security perspective because it
will no longer be able to verify or generate digital signatures.
Unless your environment does not need trust and security,
or you understand thoroughly how to provide trust and security
by other means, it is a very bad idea to compile without OpenSSL.

N.B. Compiling, or not, zxid with OpenSSL does not affect whether your
web server will use SSL or TLS. Unless you know what you are doing,
you should be using SSL at web server layer. Given that SSL is used at
web server layer, the memory footprint savings you would gain from
compiling zxid without OpenSSL may be negligible if you use dynamic
linking.

5.2.2 Compilation without libcurl
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment out the -DUSE_CURL flag from CFLAGS in Makefile and
recompile.

Disabling libcurl does not have adverse security implications: you
only loose some functionality and depending on your situation you may
well be able to live without it.

1. Without libcurl, zxid can not act as a SOAP client. This has a few
   consequences

   a. Artifact profile for SSO is not supported because it needs SOAP
      to resolve the artifact. In most cases a perfectly
      viable alternative is to use POST profile for SSO.

   b. SOAP profiles for Single Logout and NameID management (aka
      defederation) are not supported. You can use the redirect
      profiles and get mostly the same functionality.

2. Automatic CoT metadata fetching using well known
   location method is not supported without libcurl.
   You can fetch the metadata manually, e.g. using web browser,
   and place it in /var/zxid/cot directory.

   If you want to manually control your Circle of Trust
   relationships, you probably want to do this anyway so
   loss of automatic functionality may be a non-issue.<<footnote:
   If you compile with libcurl, but still want to disable
   automatic metadata fetching, investigate the ZXID_MD_FETCH
   and related configuration options.>>

3. Web Services Client (WSC) functionality is not supported
   without libcurl. Effectively this is just another case of
   "SOAP needed". If you have your own SOAP implementation,
   you may, at lesser automation, achieve much of the
   same functionality by calling the encoder and decoder
   functions manually.

5.2.3 Compiling without zlib (not supported)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

zlib is used mainly in redirect profiles. Since zlib is widely
avvailable and its foot print is small, we have made no supported
provision to compile without it. If you hack something together, let
us know.

5.3 Choosing Which Standards to Compile in (default: all)
---------------------------------------------------------

WARNING: Only regularly tested configuration is to compile all
standards in.

On space constrained systems you may shed additional weight by only
compiling in the IdM standards you actually use.  Of course, if you do
not use them, the dead function elimination should take care of them,
but sometimes you can gain additional savings in space and especially
compile time.

Another reason could be, in the land of the free, if some modules
are covered by a software patent, you may want to compile a binary
without the contested functionality.<<footnote: Please do not
ask me to add additional baggage to avoid patents. Software
patents are a plague and your efforts are best spent in getting
them overturned or changing laws so they go away.>>

You can tweak the flags, shown in accompanying table, in the Makefile
or by supplying new values in localconf.mk or on commend line. For example

  make TARGET=sol8 ENA_SAML2=0

would disable SAML 2.0 (and trigger build for Sparc Solaris 8).

<<table: Conditional inclusion of standards
Makefile flag    Standard      Comments
================ ============= ======================================
ENA_SSO=1        All SSO       Must be enabled for any of SSO to work
ENA_SAML2=1      SAML 2.0
ENA_FF12=1       ID-FF 1.2     Requires ENA_SAML11=1
ENA_SAML11=1     SAML 1.1
ENA_WSF=1        All WSF       Must be enabled for any of WSF to work
ENA_WSF2=1       ID-WSF 2.0
ENA_WSF11=1      ID-WSF 1.1
>>

5.4 localconf.mk
----------------

You can use localconf.mk to remember your own make options,
such as TARGET and different ENA flags, without editing
the distributed Makefile.

One useful option to put in localconf.mk is ENA_GEN which
will turn on the dependencies that will trigger generation
of the files in zxid/c directory. For example

  echo 'ENA_GEN=1' >>localconf.mk
  make

> WARNING: If you are confused about compilation flags
> appearing "out of nowhere", despite not being mentioned
> in the Makefile, be sure you have not inadvertently created
> a localconf.mk (perhaps you rsynced the sources from another
> machine and forgot to remove localconf.mk). A similar
> problem can occur if you accidentally copied deps.dep
> from another machine. It will reference the dependencies
> with the paths of that machine. Just remove the deps.dep
> to solve this issue.

5.5 Tough Compilation Errors
----------------------------

ZXID was mostly developed on gcc 3.4.6 and explicitly has not been
tested on 4.x series of gcc (due to deprecation of commonly useful
features in that gcc series). If you hit compile time problems with
4.x gcc, try with 3.4.x series gcc first, before reporting bugs.

***

5.6 Tough Linking Errors
------------------------

* There are some, yet unspecified, compilation or linking
  errors that manifest on 64 bit or mixed 32 and 64 bit Linux platforms.
  Current suggested workaround is to compile 32 bit only.

* Linking error that has been reported a lot on the Google for
  other software packages:

    /usr/lib/libc_nonshared.a(elf-init.oS): In function `__libc_csu_init':
    (.text+0x2a): undefined reference to `__init_array_start'
    /usr/bin/ld: zxidjava/libzxidjni.so: hidden symbol `__fini_array_end' isn't defined
    /usr/bin/ld: final link failed: Nonrepresentable section on output

  This seems to be either glibc version related or linker configuration
  related (or compiler version?). See

    http://www-gatago.com/gnu/gcc/help/43379636.html
    http://lists.debian.org/debian-68k/2006/07/msg00017.html

  The verdict seems to be that somewhere in glibc-2.3 development
  it was broken by design and nobody cared to provide any viable
  fix short of upgrading your binutils. Seems to be Ulrich Drepper
  at it again, turning symbols hidden in teeny version number upgrade.

  A symptom of this is if this command gives the follwing output

     ld --verbose | egrep '__init_array_(start|end)'
     PROVIDE_HIDDEN (__init_array_start = .);
     PROVIDE_HIDDEN (__init_array_end = .);  

  You can manually attempt to link with

    gcc -nostdlib -o zxidjava/libzxidjni.so -shared --export-all-symbols -Wl,-whole-archive -Wl,--allow-multiple-definition zxidjava/zxid_wrap.o -L. -lzxid -lpthread -L/usr/local/lib -L/usr/local/ssl/lib -lcrypto -lcurl -lz

  The trick is to supply the -nostdlib flag.

  ZXID project considers this to be glibc or binutils bug and will not
  investigate further. Either use the workaround link line or upgrade
  binutils, glibc, or both and complain to glibc authors or binutils authors.

5.7 Overview of the xsd2sg.pl Tool
----------------------------------

The xsd2sg.pl tool, distributed as part of Plaindoc system, is at the
heart of the ZXID conde generation. Unfortunately the tool is still
in flux. What follows is general outline valid as of March 2007.

xsd2sg.pl can be invoked in three ways (see xsd2sg.pl --help), but the
invocation of interest to use is the -S SG to XSD with Code generation
mode. This mode takes as inputs all .sg files and names of
decoding root elelment. It will generate one mega parser that is capable
of decoding any root element (and since root elements contain other
elements, then really all elements). The decoder works by recursive
descent parser principle so there will be smaller decoder functions to
call. Similarily the encoder is generated by creating many unit
encoders for different elements. Then the parent elements will
call the child element encoders.

5.7.1 Process .sg to create %dt hash (throw away XSD output)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

xsd2sg.pl first processes the .sg files to XSD. As a side effect it
populates %dt multidimentsional hash

  $dt{'element'}{elemname}[array-of-sublemens-and-attrs]
  $dt{'attribute'}{attrname} = "";
  $dt{'complexType'}{typename}[array-of-sublemens-and-attrs]
  $dt{'group'}{groupname}[array-of-sublemens-and-attrs]
  $dt{'attributeGroup'}{agname}[array-of-attrs]

Where the array-of-subelems-and-attrs can contain tag or reference to fundamental
or derived type, e.g.

  "tag$sa:Assertion$min$max"
  "ref$xs:anyURI$"
  "ref$sa:Assertion$"
  "base$$0$1$base-type"
  "enum$n1$n2$n3$n4$n5"
  "any$$min$max$pc$ns"
  "anyAttribute$$min$1"
  "_d$$0$1$simplecontent"

5.7.2 The expanding element definitions in %dt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since the concrete goal is to generate data structures for elements
irrespective of how types were used to describe the data, we need to
expand any type reference into element definition where it appears.
This is done by expand_element() which will in its turn call
expand_complex_type(), expand_group() and expand_attribute_group()
to do the job.

5.7.3 Generation Phase
~~~~~~~~~~~~~~~~~~~~~~

Once %dt is in expanded form, the get_element() is called for each
namespace. It will generate the code for encoders and decoders
for the elements in the namespace. Finally the root decoder
is gnerated using gen_element(). Furing generation following
template files are used

  aux-templ.c      - Code generation template for auxiliary functions
  dec-templ.c      - Code generation template for decoders
  enc-templ.c      - Code generation template for encoders
  getput-templ.c   - Code generation template for accessor functions

A lot of the generated headers are simply hard coded in the xsd2sg.pl (i.e. no
templates).

6 Net::SAML Perl Module
=======================

* perl CGI example: zxid.pl
* using with mod_perl

After building the main zxid tree, you can

  cd Net
  perl Makefile.PL
  make
  make test      # Tests are extremely sparse at the moment
  make install

This assumes you use the pregenerated Net/SAML_wrap.c and Net/SAML.pm files
that we distribute. If you wish to generate these files from origin,
you need to have SWIG installed and then say in main zxid directory

  make perlmod ENA_GEN=1  # Makes all available perl modules (including heavy low level ones)
  make samlmod ENA_GEN=1  # Only makes Net::SAML (much faster)
  make wsfmod  ENA_GEN=1  # Only makes Net::WSF (much faster)

>  WARNING: Low level interface is baroque, and consequently, it
>  will take a lot of disk space, RAM and CPU to build it: 100 MB
>  would not be exaggeration and over an hour (on 1GHz CPU). Build
>  time memory consumption of single cc1 process will be over
>  256 MB of RAM. You have been warned.

6.1 Current major modules are
-----------------------------

* Net::SAML - The high level interfaces for Single Sign-On (SSO)
* Net::SAML::Raw - Low level assertion and protocol manipulation interfaces
* Net::SAML::Metadata - Low level metadata manipulation interfaces

6.2 Planned modules
-------------------

* Net::WSF - The high level interfaces for Web Services Frameworks (WSF)
* Net::WSF::Raw - The low level interfaces for WSF variants
* Net::WSF::WSC - The high level interfaces for Web Services Clients
* Net::WSF::WSC:Raw

6.3 Perl API Adaptations
------------------------

The perl APIs were generated from the C .h files using SWIG. Generally any
C functions and constants that start by zxid_, ZXID_, SAML2_, or SAML_
have that prefix changed to <<tt: Net::SAML::>>. Note, however, that
the zx_ prefix is not stripped.

Since ZXID wants to keep strings in many places in length + data
representation, namely as ~struct zx_str~, SWIG typemaps were used
to make this happen automatically. Thus any C function that takes as
an argument <<tt: struct zx_str*>> can take a perl string
directly. Similarly any C function that returns such a pointer, will
return a perl string instead. As a final goodie, any C function, such
as

  struct zx_str* zx_ref_len_str(struct zx_ctx* c, int len, char* s);

that takes ~length~ and <<tt: s>> as explicit arguments, takes only single
argument that is a perl string (the one argument automatically satisfies
two C arguments, thanks to a type map). The above could be called like

  $a = Net::SAML::zx_ref_len_str(c, "foo");

First the "foo" satisfies both ~len~ and ~s~, and then the return value
is converted back to perl string.

6.4 Testing Net::SAML and zxid.pl as CGI script
-----------------------------------------------

To test the perl module, you must restart the mini_httpd(8) so
that it recognizes zxid.pl as CGI script:

  mini_httpd -p 8443 -c zxid.pl -S -E zxid.pem

Then start browsing from

  https://sp1.zxidsp.org:8443/zxid.pl

or if you want to avoid the common domain cookie check

  https://sp1.zxidsp.org:8443/zxid.pl?o=E

6.5 Testing Net::SAML and zxid.pl under mod_perl
------------------------------------------------

You can run zxid.pl under mod_perl using the Apache::Registry
module. See <<link:apache.html: Apache recipe>> for how
to compile Apache to support mod_perl. After configuration
it should work the same as the CGI approach.

6.6 Debugging Net::SAML with GDB
--------------------------------

As bizarre as it may sound, it is actually quite feasible to debug
libzxid and the SAML_wrap.c using GDB while in perl. For example

  cd zxid
  gdb /usr/local/bin/perl
  set env QUERY_STRING=o=E
  r ./zxid.pl

If the script crashes inside the C code, GDB will perfectly
reasonably take control, allowing you to see stack back-trace (bt)
and examine variables. Of course it helps if openssl and perl
were compiled with debug symbols (libzxid is compiled
with debug symbols by default), but even if they weren't you
can usually at least get some clue.

When preparing a perl module, generally Makefile.PL mechanism causes
the same compilation flags to be used as were used to compile the perl
itself. Generally this is good, but if libzxid was compiled with
different flags, mysterious errors can crop up. For example, I compile
my libzxid against openssl that I have also compiled myself. However, I
once had a bug where the perl had been compiled such that the Linux
distribution's incompatible openssl would be picked by perl compile
flags, resulting in mystery crashes deep inside openssl ASN.1 decoder
routines (c2i_ASN1_INTEGER() while in d2i_X509() to be exact). When I
issued `info files' in GDB I finally realized that I was using the
wrong openssl library.

7 PHP extension php_zxid.so
===========================

The PHP integration is incomplete due to incomplete support in SWIG
for php5. However, enough interface exists to get most high level API
working and thus successfully run an SP.

7.1 Building and Installing ZXID PHP extension
----------------------------------------------

After building main zxid distribution, say

  make phpzxid

You MUST have php-config(1) in path. If not, try

  make phpzxid PHP_CONFIG=/path/to/php-config

If the extension built successfully, you can use it by copying
it to a suitable place, e.g.

  make phpzxid_install

The install again uses the php-config(1) to figure out where
php(1) can find the module.

Next you need to decide whether to run under Apache mod_php
setup (apache), or as CGI (any web server).

7.1.1 Running PHP as Apache mod_php
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

See <<link:apache.html: Apache recipe>> for how
to compile Apache to support mod_php.

7.1.2 Running PHP as CGI (any web server)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the CGI case you generally make sure your CGI script starts like

  #!/usr/bin/php
  <?
  dl("php_zxid.so");  # These three lines can go to initialization: they only need to run once
  $conf = "PATH=/var/zxid/&URL=https://sp1.zxidsp.org:8443/zxidhlo.php";
  $cf = zxid_new_conf_to_cf($conf);
  ?>

The first line makes sure the file is executed with php interpreter. You should
change it to match the path where your php is installed. You also need to
make your CGI script executable, e.g:

  chmod a+x mycgi.php

Then you place the CGI script in a directory in the document tree
of the web site and make sure your http server is configured (permitted)
to execute CGI scripts in that directory.

One tricky thing that can go wrong is dynamic linking. When you compiled
the php_zxid.so module, some linking dependencies are usually created.
Problem arises if some of the dependencies are not in the paths
allowed for dynamic linking by your web server. The paths allowed
by web server can easily be different than in your shell and some
web servers even ignore ~LD_LIBRARY_PATH~ environment variable. Sometimes
you just have to copy the dependency libraries to one of the allowed
directories. This is "dirty", but works. See ldconfig(8) and
section <<see:ZXID-Installing-CannedTutorialRunningZXIDasCGIundermini_httpd-AccessingZXID>> for further information.

You can easily see the dependencies using ldd(1)

  ldd /apps/php/5.1.6/lib/php/extensions/no-debug-non-zts-20050922/php_zxid.so
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb7796000)
        libdl.so.2 => /lib/libdl.so.2 (0xb7792000)
        libcurl.so.3 => /apps/lib/libcurl.so.3 (0xb7611000)
        libz.so.1 => /apps/lib/libz.so.1 (0xb75fe000)
        libc.so.6 => /lib/libc.so.6 (0xb74dd000)
        /lib/ld-linux.so.2 (0x80000000)

In this example the suspect library dependencies are
/apps/lib/libcurl.so.3 and /apps/lib/libz.so.1 because they are
outside normal places, i.e. /lib and /usr/lib.

Another thing to remember is that CGI specification requires that the
~Content-Type~ header and an empty line (to separate headers from
content) is emitted before the actual page. If this happens, the
page will mysteriously not appear although your script ran successfully.

Sometimes the debug output can end up in stdout (e.g. somewhere stderr
was redirected to stdout) and this will garble the returned web page.
Easy fix is to disable debug output by not supplying ~ZXID_AUTO_DEBUG~.
See section <<see:ZXID-ZXID_simpleAPI-HelloWorld-AUTOoptions>>.

7.2 Programming with ZXID PHP Extension
---------------------------------------

To use the ZXID PHP extension you must add near beginning of your script

  dl("php_zxid.so");   // Load the module

You may need to tweak the paths, or ~LD_LIBRARY_PATH~, to get this to work.

After this, you can use the PHP interface much the same way as you
would use the C interface. See the distributed zxid.php and
zxidhlo.php for further usage examples.

8 Python Extension
==================

TBD using SWIG

9 Java Native API (JNI): zxidjava package and zxidjni class
===========================================================

9.1 Building the JNI
--------------------

After building main zxid distribution, say

  make javazxid

You must have set ~JNI_INC~ variable correctly in the Makefile (or in
localconf.mk) and ~javac~ must be in path (or you must set ~JAVAC~
variable). For the servlet or Tomcat support, you must make sure
~SERVLET_PATH~ points to your servlet-api.jar file. The ZXID Java
interface has been mainly tested with ~j2sdk1.4.2~ and some versions
of Java SDK 1.5.

If you have done changes that require regeneration of the
zxidjava/zxid_wrap.c file you should build with

  make javazxid ENA_GEN=1

but this requires that you have swig(1) installed. Depending
on the changes it may also require xsd2sg.pl and gperf(1),
see "Compilation for Experts" section, above, for full
explanation. As of January 2007, all of the Java JNI interface
is swig(1) generated - there are no human authored files. However,
we anticipate building a helper Java library to facilitate
use of the JNI - contributions welcome.

After compilation, just copy the class files and the libzxidjni.so
to suitable locations in your system (Makefile lacks any specific
Java installation target because the author has not yet made up his
mind about what makes sense). When you run Java programs that
use the zxidjni class, you must makes sure the libzxidjni.so is
found by the dynamic linker - usually this means setting ~LD_LIBRARY_PATH~
environment variable. The zxid-java.sh shell script demonstrates
how to do this for the example CGI program zxid.java.

9.1.1 MacOS X: JNI Notes
~~~~~~~~~~~~~~~~~~~~~~~~

* The resulting library is called libzxidjni.jnilib
* You may need to supply additional arguments to
  java command:

    java -classpath .:zxidjava -Djava.library.path=zxidjava zxid

* On Mac, it seems ~export LD_LIBRARY_PATH=zxidjava~ is not needed.


9.2 Programming with ZXID Java API
----------------------------------

For detailed usage examples of the Java interface you should study
the zxid.java file.

The Java interface is contained in a package called ~zxidjava~. This
package contains the main wrapper class ~zxidjni~ as well as a number
of data type specific classes. The ~zxidjni~ class is just
a container for procedural zxid API - all methods of this class
are static.

To start using the ZXID Java interface you need to do two
things:

  import zxidjava.*;

somewhere near top of your program pulls in the zxidjava package,
including the zxidjni class. Then you need to have a static
initializer smewhere in your program to pull in the libzxidjni.so:

  public class myprog {
    static {
      System.loadLibrary("zxidjni");
    }

    public static void main(String argv[]) throws java.io.IOException
    {
      // ...
    }
  }

From here on you can call the C API procedures as static methods
of the zxidjni class, e.g:

  cf = zxidjni.new_conf("/var/zxid/");

Note that the ~zxid_~ prefix is omitted in favour of the ~zxidjni~
class name qualifier.

9.3 Known Problems and Limitations
----------------------------------

The zx_str type is generally NOT nul terminated. We try to
map these in the SWIG type maps, but any function returning
char* currently maps to Java String type, yet there is no
way of knowing how long the string type is. Therefore
it's not safe to call functions returning char*. For example,
consider

  int zx_LEN_SO_sa_Action(struct zx_ctx* c, struct zx_sa_Action_s* x);
  char* zx_ENC_SO_sa_Action(struct zx_ctx* c, struct zx_sa_Action_s* x, char* p);
  struct zx_str* zx_EASY_ENC_SO_sa_Action(struct zx_ctx* c, struct zx_sa_Action_s* x);

The intent of the LEN_SO plus ENC_SO pair is that you first compute
length, allocate sufficient buffer, and then render the encoding into
the buffer. The ENC_SO in fact returns char* one past the end of the
string. It is NOT safe to cal ENC_SO from Java because the SWIG
generated interface would make Java believe that the char* one past
end of string is a C string in its own right. Thus the only
safe one to call is the EASY_ENC_SO variant.

9.4 Running as servlet under Tomcat
-----------------------------------

ZXID distribution contains subdirectory called ~servlet~. You should
link this into webapps directory of Tomcat servlet container

  cd ~/apache-tomcat-5.5.20/webapps
  ln -s ~/zxid-0.17/servlet zxidservlet

You also need to set ~allowLinking~ flag in 
apache-tomcat-5.5.20/conf/context.xml (the ~reloadable~
flag avoids having to restart Tomcat if you recompile
the .class file):

  <Context allowLinking="true" reloadable="true">...

The file servlet/WEB-INF/web.xml describes the example zxid
application. The actual application lives in servlet/WEB-INF/classes
which is actually just a symlink back to the top level of the ZXID
distribution. Therefore the zxidhello.class file appears on the top
level and the wrapper classes, which are scoped in ~zxidjava~ package,
appear in zxidjava/ subdirectory. From the servlet container's
perspective the directory appears to be
apache-tomcat-5.5.20/webapps/zxidservlet/WEB-INF/classes/zxidjava

After ~make javazxid~ and restart of Tomcat (~killall java;
apache-tomcat-5.5.20/bin/startup.sh~), you can access the application
using URL (defined in servlet/WEB-INF/web.xml)

  http://sp1.zxidsp.org:8180/zxidservlet/zxidHLO?o=E

9.5 Troubleshooting class loader
--------------------------------

If you get

  java.lang.ClassNotFoundException: zxidhlo
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1355)
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1201)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        java.lang.Thread.run(Thread.java:595)

Then you forgot to turn on the symlinks (see ~allowLinking~, above).

If you get

  java.lang.UnsatisfiedLinkError: no zxidjni in java.library.path
        java.lang.ClassLoader.loadLibrary(ClassLoader.java:1517)
        java.lang.Runtime.loadLibrary0(Runtime.java:788)
        java.lang.System.loadLibrary(System.java:834)
        zxidhlo.<clinit>(zxidhlo.java:20)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        java.lang.reflect.Constructor.newInstance(Constructor.java:274)
        java.lang.Class.newInstance0(Class.java:308)
        java.lang.Class.newInstance(Class.java:261)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        java.lang.Thread.run(Thread.java:534)

Then it is not finding zxidjava/libzxidjni.so. Either say

  export LD_LIBRARY_PATH=~/zxid-0.14/zxidjava:$LD_LIBRARY_PATH

or place libzxidjni.so in $CATALINA_HOME/shared/lib. Note that the
library name really is libzxidjni.so despite the misleading exception
suggesting just "zxidjni".

If you get

  java.lang.UnsatisfiedLinkError: Native Library /home/sampo/zxid/zxidjava/libzxidjni.so already loaded in another classloader
        java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1551)
        java.lang.ClassLoader.loadLibrary(ClassLoader.java:1511)
        java.lang.Runtime.loadLibrary0(Runtime.java:788)
        java.lang.System.loadLibrary(System.java:834)
        zxidhlo.<clinit>(zxidhlo.java:20)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        java.lang.reflect.Constructor.newInstance(Constructor.java:274)
        java.lang.Class.newInstance0(Class.java:308)
        java.lang.Class.newInstance(Class.java:261)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        java.lang.Thread.run(Thread.java:534)

Then ... ? Currently it seems you have to restart Tomcat.

If you get

java.lang.NoClassDefFoundError: javax/servlet/http/HttpServlet
        java.lang.ClassLoader.defineClass1(Native Method)
        java.lang.ClassLoader.defineClass(ClassLoader.java:620)
        java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
        java.net.URLClassLoader.access$100(URLClassLoader.java:56)
        java.net.URLClassLoader$1.run(URLClassLoader.java:195)
        java.security.AccessController.doPrivileged(Native Method)
        java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
        java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1270)
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1201)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        java.lang.Thread.run(Thread.java:595)

Then the problem is with class path. Apparently running the startup.sh script from anywhere
else than top level of Tomcat distribution produces the above error.

9.6 Logging and Debugging Tips
------------------------------

In case you add debug prints, the stderr (System.err) output
appears to go by default to apache-tomcat-5.5.20/logs/catalina.out

9.7 Debugging libzxidjni.so under jdb and gdb
---------------------------------------------

Debugging the JNI C code would appear to require running java or jdb
under gdb and setting break points in the C code. Unfortunately this
appears to be particularly tricky. A possible approach is to introduce
a sleep(1) in the C code and then use gdb to attach to the java
process. Unfortunately even this method does not seem to allow us to
set break points.

  export LD_LIBRARY_PATH=zxidjava:$LD_LIBRARY_PATH
  export QUERY_STRING='e=https%3A%2F%2Fidp.symdemo.com%3A8880%2Fidp.xml&l2=+Login+%28SAML20%3APOST%29+&fc=1&fn=prstnt&fq=&fy=&fa=&fm=exact'

N.B. In following "$" means Unix shell prompt, "%" gdb prompt, and ">"
jdb prompt.

  $ gdb jdb
  % set env LD_LIBRARY_PATH=zxidjava
  % set env QUERY_STRING=e=https%3A%2F%2Fidp.symdemo.com%3A8880%2Fidp.xml&l2=+Login+%28SAML20%3APOST%29+&fc=1&fn=prstnt&fq=&fy=&fa=&fm=exact
  % r zxid
  > stop at zxid:24
  > run
  > next      # or step
  > print cf
  > cont

10 zxid_simple() API
====================

The ZXID library provides two main APIs: the simple and the full. The
simple API is meant to get you going with minimal understanding about
SAML or SSO. It tries to encapsulate as much control flow as
possible.

10.1 Hello World
----------------

Consider the follwing "Hello World" CGI example in C<<footnote: zxidhlo.c
in the source distribution is an actually usable example program. You should
also look at its cousins zxidhlo.php and zxidhlo.pl>>
(the zxid_simple() API is available in all language bindings):

  01 #include <zx/zxid.h>
  02 #define CONF "PATH=/var/zxid/&URL=https://sp1.zxidsp.org:8443/zxid"
  03 void main() {
  04   char* res = zxid_simple(CONF, 0, 255);
  05   switch (res[0]) {
  06   case 'd': /* Logged in case */
  07     my_parse_ldif(res);
  08     my_render_content();
  09     exit(0);
  10   default:
  11     ERR("Unknown zxid_simple() response(%s)", res);
  12   }
  13 }

What happens here:

1. The CGI script calls zxid_simple() to handle SAML protocol
   according to the configuration

2. The last argument with value 255 tells zxid_simple()
   to automatically handle redirections, login screen and
   any other protocol interaction needed to make SSO happen.

3. If zxid_simple() returns, we have either succeeded in SSO
   or we have failed (all other cases are handled internally
   by zxid_simple() which calls exit(2) so it never returns).

4. In the success case, zxid_simple() returns an LDIF
   entry (as a nul terminated C string) describing the SSO and
   the attributes received. For example

     dn: idpnid=Pa45XAs2332SDS2asFs,affid=https://idp.demo.com/idp.xml
     objectclass: zxidsession
     affid: https://idp.demo.com/idp.xml
     idpnid: Pa45XAs2332SDS2asFs
     authnctxlevel: password
     sesid: S12aF3Xi4A
     cn: Joe Doe

   where

   dn:: LDAP distinguished name (part of LDIF format). Always first.
   objectclass:: Part of LDIF format.
   affid:: Specifies which IdP was used for SSO
   idpnid:: The federated ID, or pseudonym (IdP assigned NameID)
   authnctxlevel:: Rough indication of how IdP authenticated user
   sesid:: Session ID, as may be stored in cookie or used for
       file name in the session cache (/var/zxid/ses)
   cn:: Common Name. This attribute just exemplifies how any
       additional attributes the IdP may have set will appear.
       Typically the LDAP attribute names are used.

   The ~dn~ line will always be the first. All other lines may appear
   in any order. String representation of LDIF was chosen
   as it is easy to parse in most programming languages.

10.1.1 Configuration Options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The zxid_simple() can be configured in following ways (later
ways can override earlier ones).

1. Built in default configuration
2. Configuration file, usually /var/zxid/zxid.conf, if any
3. Configuration string passed as first argument.
   While configuration string can override all
   other options (i.e. it is processed last), the
   PATH specification is parsed before the configuration
   file is looked up.

Turns out that often the default configuration modified
by the configurations string is all you need - you do not
need to prepare configuration file.

See section "Configuring and Running" for complete list of
configuration options, but generally it is
sufficient to specify only a handful:

PATH:: Where files are kept and configuration file is found.
URL:: The URL of the SP
CDC_URL:: The Common Domain URL (optional, if omitted
    the Common Domain Cookie processing is disabled)

10.1.2 AUTO options
~~~~~~~~~~~~~~~~~~~

The ~auto_flags~ argument allows you to control which
operations should be performed automatically and
which should be passed to the calling application,
see "Gaining More Control" section, below, for
full description of this case.

The auto options can be added together.

<<table: zxid_simple() AUTO options
Dec  Hex    Symbol              Description
==== ====== =================== ================================================
   1   0x01 ZXID_AUTO_EXIT      Call exit(), 0=return "n", even if auto CGI
   2   0x02 ZXID_AUTO_REDIR     Automatically handle redirects, assume CGI (calls exit(2))
   4   0x04 ZXID_AUTO_SOAPC     SOAP response handling, content gen
   8   0x08 ZXID_AUTO_SOAPH     SOAP response handling, header gen
  16   0x10 ZXID_AUTO_METAC     Metadata response handling, content gen
  32   0x20 ZXID_AUTO_METAH     Metadata response handling, header gen
  64   0x40 ZXID_AUTO_LOGINC    IdP select / Login page handling, content gen
 128   0x80 ZXID_AUTO_LOGINH    IdP select / Login page handling, header gen
 256  0x100 ZXID_AUTO_MGMTC     Management page handling, content gen
 512  0x200 ZXID_AUTO_MGMTH     Management page handling, header gen
1024  0x400 ZXID_AUTO_FORMF     In idp list and mgmt screen, generate form fields
2048  0x800 ZXID_AUTO_FORMT     In idp list and mgmt screen, wrap the output in <form> tag.
4095  0xfff ZXID_AUTO_ALL       Enable all automatic CGI behaviour.
4096 0x1000 ZXID_AUTO_DEBUG     Enable debugging output to stderr.
>>

If the AUTO_REDIR flag is true, the LOCATION header is sent to stdout
and exit(0) may be called, depending on the AUTO_NOEXIT flag.

The SOAP, META, LOGIN, and MGMT flags follow convention:

  HC
  00  No automation. Only action letter is returned ("e"=login, "b"=meta, etc.)
  01  Content, not wrapped in headers, is returned
  10  Headers and content is returned
  11  Headers and content are sent to stdout, CGI style and
      exit(0) may be called, depdending on AUTO_EXIT.

Whether exit(0) is called in 11 case depends on ZXID_AUTO_NOEXIT flag.

How much HTML is generated for Login page and Mgmt page in 01 (content only) mode
depends on AUTO_PAGE and AUTO_FORM flags

  TF
  00  reserved / nothing is generated
  01  Only form fields (but not form tag itself) are generated.
  10  Complete form is generated
  11  Whole page is generated (best support)

10.1.3 Configuration options for customizing HTML
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When whole page is generated, some templating information is taken
from the configuration.

IDP_SEL_START:: All the HTML before <form> tag. This can include HTML headers
    and the <body> tag, as well as beginning of the page, allowing
    for complete color selection, stylesheet embedding, and general
    branding of the page.

IDP_SEL_NEW_IDP:: The HTML fragment to allow login using a new IdP (Auto
    CoT using IdP URL). Set to empty to hide this possibility.

IDP_SEL_OUR_EID:: Message displaying SP Entity ID, in case (technically
    minded) user needs to know this to establish relationship with an IdP.

IDP_TECH_USER:: Technical parameters that user might want to set, and
    typically would be allowed to set. May be hidden (not user
    controllable) or visible.
    fc:: Create federation (AllowCreate flag)
    fn:: Name ID format
        prstnt:: Persistent (pseudonym)
        trnsnt:: Transient, temporary pseudonym

IDP_TECH_SITE:: Technical parameters that the site administrator
    should decide and set. Usually hidden fields:
    fq:: Affiliation ID (usually empty)
    fy:: Consent obtained by SP for the federation or SSO
        empty:: No statement about consent
        urn:liberty:consent:obtained:: Has been obtained (unspecified way)
        urn:liberty:consent:obtained:prior:: Obtained prior to present
            transaction, e.g. user signed terms and conditions of service
        urn:liberty:consent:obtained:current:implicit:: Consent
            is implicit in the situation where user came to invoke service
        urn:liberty:consent:obtained:current:explicit:: Obtained explicitly
        urn:liberty:consent:unavailable:: Consent can not be obtained
        urn:liberty:consent:inapplicable:: Obtaining consent is not
            relevant for the SP or service.
    fa:: Authentication Context (strength of authentication) needed by the SP
    fm:: Matching rule for authentication strength (usually empty, IdP decides)
    fp:: Forbid IdP from interacting with the user (IsPassive flag)
    ff:: Request reauthentication of user (ForceAuthn flag)

10.2 Gaining More Control
-------------------------

The fully automated interface works well for CGI deployment
but probably is not appropriate in more complex situations.
You can use zxid_simple() without automation and prevent
it from accessing the system layer too directly. Consider

  01 #define CONF "PATH=/var/zxid/&URL=https://sp1.zxidsp.org:8443/zxid"
  02 int main() {
  03   char* res;
  04   res = zxid_simple(CONF, getenv("QUERY_STRING"), 0);
  05   switch (res[0]) {
  06   case 'L': printf("%s", res); exit(0);  /* Redirect */
  07   case 'C': printf("%s", res); exit(0);  /* Content-type + content */
  08   case '<': printf("Content-Type: text/html\r\n\r\n%s", res); exit(0);
  09   case 'n': exit(0);
  10   case 'b': my_send_metadata();
  11   case 'e': my_render_login_screen();
  12   case 'd': /* Logged in case */
  13     my_parse_ldif(res);
  14     my_render_content();
  15     exit(1);
  16   default:
  17     ERR("Unknown zxid_simple() response(%s)", res);
  18   }
  19 }

Here we specify zero as ~auto_flags~, thus all automation is
disabled. This means that the res can take more varied
shape and the calling application has to be prepared
to handle them. The different cases are distinguished by
the first character of the res string:

L:: Redirection request (L as in Location header). The
    full contents of the res is the redirection request,
    ready to be printed to stdout of a CGI. If you want
    to handle the redirection some other way, you can
    parse the string to extract the URL and do your thing.
    This res is only returned if you did not
    set ZXID_AUTO_REDIR.

    Example:

      Location: https://sp1.zxidsp.org:8443/zxid?o=C

C:: Content with Content-type header. The res is ready
    to be printed to the stdout of a CGI, but if you
    want to handle it some other way, you can parse
    the res to extract the header and the actual body.

    Example:

      CONTENT-TYPE: text/html
      
      <title>Login page</title>
      ...

    Example (metadata):

      CONTENT-TYPE: text/xml
      
      <m:EntityDescriptor>
      ...

Less than ("<"):: Content without headers. This could
    be HTML content for login page or metadata XML.
    To know which (and set content type correctly),
    you would have to parse the content. This res
    format is only applicable if you did not specify
    ZXID_AUTO_CTYPE (but did specify ZXID_AUTO_CONTENT).

n:: Do nothing. The operation was somehow handled internally
    but the exit(2) was not called (e.g. ZXID_AUTO_SOAP
    was NOT specified). The application should NOT attempt
    generating any output.

b:: Indication that the application should send SP
    metadata to the client. This res is only returned
    if you did not set ZXID_AUTO_META.

e:: Indication that the application should display the
    login page. Application may use zxid_idp_list()
    to render the IdP selection area. This res is only returned
    if you did not set ZXID_AUTO_CONTENT.

d:: Indication that SSO has been completed or that there
    was an existing valid session in place. The res is an
    LDIF entry containing attributes that describe the
    SSO or session. See "Hello World" section for
    description of the LDIF data.

    Usually your application would parse the attributes
    and then render its application specific content.
    If you want to render the SSO management form (with
    logout and defederate buttones), you can call
    zxid_fed_mgmt().

Asterisk ("*"):: Although any unknown letter should be
    interpreted as an error, we follow convention
    of prefixing errors with an asterisk ("*").

10.3 Some Generalization and Optimizations
------------------------------------------

The simplest APIs are easy to use and suitable for CGIs where the
program is restarted anew every time. However in situations where the
script engine stays alive persistently, it is wasteful to reparse (and
reallocate) the configuration every time. Consider following PHP
snippet designed to be used with mod_php:

  01 # Put this in the PHP initialization (it only needs to run once)
  02 dl("php_zxid.so");
  03 $conf = "PATH=/var/zxid/&URL=https://sp1.zxidsp.org:8443/zxiddemo.php";
  04 $cf = zxid_new_conf_to_cf($conf);
  05 <?   # For every page that is accessed
  06 $qs = $_SERVER['REQUEST_METHOD'] == 'GET'
  07       ? $_SERVER['QUERY_STRING']
  08       : file_get_contents('php://input');
  09 $res = zxid_simple_cf($cf, -1, $qs, null, 0x1800);
  10 switch (substr($res, 0, 1)) {
  11 case 'L': header($res); exit;  # Redirect
  12 case 'n': exit;   # already handled
  13 case 'b': my_send_metadata();
  14 case 'e': my_render_login_screen();
  15 case 'd': break;  # Logged in case -- fall through
  16 default:  error_log("Unknown zxid_simple() res(%s)", res); exit;
  17 }
  18 # *** Parse the LDIF in $res into a hash of attributes (see zxidhlo.php)
  19 
  20 ?>
  21 <html><title>Protected content, logged in as <$=$attr['cn']$></title>
  22 ...
  23 </html>
  24 <?
  25 function my_render_login_screen()
  26 {
  27 ?>
  28 <html><title>Please Login Using IdP</title>
  29 ...
  30 <?=zxid_idp_select_cf($cf, 0, 0x1800)?>
  31 ...</html>
  32 <? exit; }?>

Notes

1. Line 4 creates a system-wide configuration object that is later
   used by the other API calls
2. On line 9 we call zxid_simple_cf() with the created object. The
   second and third argument specify a buffer or string that contains
   the CGI form data to parse. This may come from ~QUERY_STRING~ of a
   GET request or from HTTP body of a POST request, as determined
   on line 8. The -1 means the length of the data should be
   determined using strlen(3), i.e. C string nul termination.
   The ~auto_flags == 0x1800~ enables form tag wrapping and debug
   prints, but otherwise automation is disabled.
3. Since automation was disabled, we need to handle several
   cases of possible outcomes from zxid_simple_cf(), on lines 10-17.
4. From line 18 onwards we handle the login successful or already
   logged in case. First we split the LDIF entry into a hash
   so that we can access the attributes easily (e.g. ~cn~ on line 20).
5. On line 30 we call zxid_idp_list_cf() to create the form
   for choosing which IdP to use for login (remember that
   ~auto_flags == 0xc0~ enabled the form wrapper). As can be
   seen the same configuration object, ~$cf~, is used through out.

10.4 Java Servlet Example Using Tomcat
--------------------------------------

Consider

  01 import zxidjava.*;
  02 import java.io.*;
  03 import javax.servlet.*;
  04 import javax.servlet.http.*;
  05 public class zxidhlo extends HttpServlet {
  06   static { System.loadLibrary("zxidjni"); }
  07   static final String conf
  08     = "PATH=/var/zxid/&URL=http://sp1.zxidsp.org:8080/zxidservlet/zxidHLO";
  09   public void do_zxid(HttpServletRequest req, HttpServletResponse res, String qs)
  10                       throws ServletException, IOException {
  11     String ret = zxidjni.simple(conf, qs, 0xd54);
  12     switch (ret.charAt(0)) {
  13     case 'L':  /* Redirect: ret == "LOCATION: urlCRLF2" */
  14       res.sendRedirect(ret.substring(10, ret.length() - 4));
  15       return;
  16     case '<':
  17       switch (ret.charAt(1)) {
  18       case 's':  /* <se:  SOAP envelope */
  19       case 'm':  /* <m20: metadata */
  20         res.setContentType("text/xml");
  21         break;
  22       default:
  23         res.setContentType("text/html");
  24       break;
  25       }
  26       res.setContentLength(ret.length());
  27       res.getOutputStream().print(ret);
  28       break;
  29     case 'd': /* Logged in case */
  30       //my_parse_ldif(res);
  31       res.setContentType("text/html");
  32       res.getOutputStream().print(zxidjni.fed_mgmt(conf, 0xd54));
  33       break;
  34     default:
  35       System.err.print("Unknown zxid_simple() response:");
  36       System.err.print(ret);
  37     }
  38   }
  39   public void doGet(HttpServletRequest req, HttpServletResponse res)
  40                     throws ServletException, IOException {
  41     // LECP/ECP PAOS header checks
  42     do_zxid(req, res, req.getQueryString());
  43   }
  44   public void doPost(HttpServletRequest req, HttpServletResponse res)
  45                      throws ServletException, IOException {
  46     String qs;
  47     int len = req.getContentLength();
  48     byte[] b = new byte[len];
  49     int got = req.getInputStream().read(b, 0, len);
  50     qs = new String(b, 0, got);
  51     do_zxid(req, res, qs);
  52   }
  53 }


10.5 Shell Script API
---------------------

Any Bourne shell (Unix shell) shell script can be converted
to a SAML SSO enabled CGI script using zxidsimple(1) helper
utility. The helper utility simply wraps the zxid_simple()
API function so that the inputs can be provided as arguments,
or in case of ~qs~ as stdin, and the output is returned on stdout.

Synopsis

   zxidsimple -o ldif CONF AUTO_FLAGS <cgi-input

Typical usage (see also zxidhlo.sh):

  CONF="PATH=/var/zxid/&URL=https://sp1.zxidsp.org:8443/zxidhlo.sh"
  ./zxidsimple -o /tmp/zxidhlo.sh.$$ $CONF 4094 || exit;
  IFS="
  "
  res=`cat /tmp/zxidhlo.sh.$$`
  case "$res" in
  dn*)
    for x in $res; do
      case "$x" in
      sesid:*)  SID=${x##*sesid: } ;;
      idpnid:*) NID=${x##*idpnid: } ;;
      cn:*)     CN=${x##*cn: } ;;
      esac
    done
    ;;
    *) echo "ERROR($res)" >>/tmp/hlo.err; exit ;;
  esac

  cat << EOF
  Content-Type: text/html

  <title>ZXID HELLO SP Mgmt</title>
  <h1>ZXID HELLO SP Management (user $CN logged in, session active)</h1>
  <form method=post action="zxidhlo.sh?o=P">
  <input type=hidden name=s value="$SID">
  <input type=submit name=gl value=" Local Logout ">
  <input type=submit name=gr value=" Single Logout (Redir) ">
  </form>
  EOF

The zxidsimple(1) utility will return exit value 1 if it
handled a SAML protocol operation (by outputting to stdout
whatever was appropriate). The shell script
should not do any further processing and just exit.

If the exit value is 0 (success) then SSO has been done. Since the
attributes from the SAML assertion are usually interesting, you can
capture them to a temporary file using the -o option.

First we split the result of the backtick into a list on (literal)
newline. Then we process the list with for loop and look with case
for the interesting attributes and capture them into local variables.

Finally the protected content page is output.

10.5 Form Field Naming
----------------------

The ZXID cgi interface assumes certain hardwired form field
names. These are not configurable (and there is no intent to make them
configurable). The cgi fields may appear either in query string (GET
method) or as POST content (though depending on your programming
environment and language, you may need to read the POST data in
yourself prior to calling zxid_simple()).

10.5.1 Common Fields
~~~~~~~~~~~~~~~~~~~~

o:: Operation. In particular o=P means that form uses POST method.
s:: Session ID
RelayState:: SAML 2.0 mandated field name for relay state
SAMLart:: SAML 2.0 mandated field name for SAML artifact
SAMLResponse:: SAML 2.0 mandated field name for SAML response, especially in POST profile
SAMLRequest:: SAML 2.0 mandated field name for SAML request
SigAlg:: SAML 2.0 mandated field name for signature algorithm in redirect binding
Signature:: SAML 2.0 mandated field name for signature in redirect binding

10.5.2 IdP Selection (Login) Screen
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

u:: User (local login)
p:: Password (local login)
c:: Common Domain Cookie
e:: Entity ID (manual entry field)
d:: Entity ID (from popup or radio box)
i:: Protocol index
l1:: Login using artifact profile (same as i=1)
l2:: Login using POST profile (same as i=2)
l1EID:: Login using specified IdP (artifact profile), same as e=EID&i=1
l2EID:: Login using specified IdP (POST profile), same as e=EID&i=2
fc:: Allow Create flag
fp:: IsPassive flag
ff:: Force Authentication flag
fn:: NameID format
fq:: Affiliation ID
fy:: Consent field
fm:: Matching rule
fa:: Authentication Context Class
fr:: Relay State

The IdP selection form (aka Login) screen can be implemented, using
the above documented form interface, in many ways as following
examples illustrate.

*Example IdP Selection Form: Popup menu method*

***

*Example IdP Selection Form: Separate IdP buttons method*

  <form method=post
    action="http://sp1.zxidsp.org:8080/zxidservlet/zxidHLO?o=P">
  <h3>Login Using New IdP</h3>
  <p>IdP URL <input name=e size=80>
      <input type=submit name=l1 value=" Login (A2) ">
      <input type=submit name=l2 value=" Login (P2) "><br>

  <h3>Login Using Known IdP</h3>
  <input type=submit name="l1https://a-idp.liberty-iop.org:8881/idp.xml"
         value=" Login to https://a-idp.liberty-iop.org:8881/idp.xml (A2) ">
  <input type=submit name="l2https://a-idp.liberty-iop.org:8881/idp.xml"
         value=" Login to https://a-idp.liberty-iop.org:8881/idp.xml (P2) ">

  <h3>Technical options</h3>
  <input type=checkbox name=fc value=1 checked> Create federation,
     NID Format: <select name=fn>
                   <option value=prstnt>Persistent
                   <option value=trnsnt>Transient
                   <option value="">(none)
                 </select><br>

  <input type=hidden name=fq value="">
  <input type=hidden name=fy value="">
  <input type=hidden name=fa value="">
  <input type=hidden name=fm value="">
  <input type=hidden name=fp value=0>
  <input type=hidden name=ff value=0>
  </form>

*Example IdP Selection Form: IdP links method*

***

10.5.3 Single Logout and Federation Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

gl:: Local Logout
gr:: Single Logout using redirection
gs:: Single Logout using SOAP
gt:: NameID Managment (redirect)
gu:: NameID Management (SOAP)

*Example Management Form*

  <form method=post action="zxid?o=P">
  <input type=hidden name=s  value="!!sid">
  <input type=submit name=gl value=" Local Logout ">
  <input type=submit name=gr value=" Single Logout (Redir) ">
  <input type=submit name=gs value=" Single Logout (SOAP) ">
  <input type=submit name=gt value=" Defederate (Redir) ">
  <input type=submit name=gu value=" Defederate (SOAP) ">
  </form>

11 Integration with Existing Web Sites
======================================

Single Sign-On is used to protect some useful resources. ZXID does not
have any means of serving these resources, rather a normal web server
or application server should do it. ZXID should just concentrate on
verifying that a user has valid session, and if not, establishing the
session by way of SSO.

11.1 Brief Overview of Control Flow
-----------------------------------

The SAML 2.0 specifications mandate a wire protocol, and in order
to speak the wire protocol, the SP application typically
has to follow certain standard sequence of control flow.

<<dia: sp-flow,,:bg: Typical control flow of ZXID SP>>

First a user<<footnote: The user is often referred to as "Principal" in
more technical jargon. Although the human user and web browser are
distinct entities, we do not stress that separation here. Whatever
user "does" really will, in protocol, appear as web browser sending
requests.>> tries to access a web site that acts in SP role. This
triggers following sequence of events

1.  User is redirected to URL in a common domain. This is so that
    we can read the Common Domain Cookie that indicates which
    IdP the user uses. Alternatively, if you started at
    https://sp1.zxidsp.org:8443/zxid?o=E, the CDC check is
    by-passed and flow 2b. happens.

2.  After the CDC check, a Authentication Request (AuthnReq) is
    generated. The IdP may have been chosen automatically
    using CDC (2a), or there may have been some user interface
    interaction (not show in the diagram) to choose the IdP.

3.  User is redirected to the IdP. The redirection carries
    as a query string a compressed and encoded form of
    the SAML 2.0 AuthnReq.

4.  Once the IdP has authenticated the user, or observed
    that there already is a valid IdP session (perhaps from
    a cookie), the IdP redirects the user back to the SP.

    The AuthnResponse may be carried in this redirection
    in a number of alternate ways

    a. The redirect contains a special token called
       +artifact+. The artifact is a reference to the
       AuthnResponse and the SP needs to get the
       actual AuthnResponse by using a SOAP call (the
       4bis step).

    b. The "redirect" is actually a HTML page with
       a form and little JavaScript that causes the
       form to be automatically posted to the SP.
       The AuthnResponse is carried as a form field.

5.  After verifying that AuthnResponse indicated a
    success, the SP establishes a local session for the
    user (perhaps setting a cookie to indicate this).
    
    Depending on how the SP to web site integration
    is done the user is taken to the web site in
    one of the two ways

    a. Redirect to the content. This time the session
       is there, therefore the flow passes directly
       from check session to the web content.

    b. It is also possible to show the content directly
       without any intervening redirection.

11.2 Redirect Approach to Integration
-------------------------------------

11.3 Pass-thru Approach to Integration
--------------------------------------

11.3.1 mod_perl pass-thru
~~~~~~~~~~~~~~~~~~~~~~~~~

11.3.2 PHP pass-thru
~~~~~~~~~~~~~~~~~~~~

11.3.3 mod_zxid pass-thru
~~~~~~~~~~~~~~~~~~~~~~~~~

11.4 Proxy Approach to Integration
----------------------------------

12 Full Native C API
====================

The generated aspects of the native C API are in c/*-data.h, for example

  c/zx-sa-data.h

Studying this file is very instructive.<<footnote: emacs tip: run
`make tags' and then try hitting M-. while cursor is over a struct
or function name in c/zx-sa-data.h - this makes navigation painless.>>

12.1 C Data Structures
----------------------

From .sg a header (NN-data.h) is generated. This header contains structs that
represent the data of the elements. Each element and attribute
generates its own node. Even trivial nodes like strings have to be
kept this way because the nodes form basis of remembering the ordering
of data. This ordering is needed for exclusive XML canonicalization,
and thus for signature verification.<<footnote: It's unfortunate that
the XML standards do not make this any easier. Without order
maintenance requirement, it would be possible to represent trivial
child elements directly as struct fields. An approach that tried to do
just this is available from CVS tag GEN_LALR (ca. 29.5.2006).>>

Any missing data is represented by NULL pointer.

Any repeating data is kept as a linked list, in reverse order of being
seen in the data stream.<<footnote: Reverse order is just an
optimization - or an artifact of simply adding latest element to the
head of the list. If this bothers you, it's easy enough to reverse the
list afterwards. Linked list is simple and works well for data whose
order does not matter much (we use separate pointer for remembering
the canonicalization order) and where random access is not needed, or
cardinality is low enough so that simple pointer chasing is efficient
enough.>>

Simple elements and all attributes are represented by simple string node
(even if they are booleans or integers).

*Example*

Consider following XML

  <ds:Signature>
     <ds:SignedInfo>
       <ds:CanonicalizationMethod
           Algorithm="http://w3.org/xml-exc-c14n#"/>
       <ds:SignatureMethod
           Algorithm="http://w3.org/xmldsig#rsa-sha1"/>
       <ds:Reference
           URI="#RrcrNwFIw6n">
         <ds:Transforms>
           <ds:Transform
               Algorithm="http://w3.org/xml-exc-c14n#"/>
           <ds:Transform
               Algorithm="http://w3.org/xmldsig#env-sig"/></>
         <ds:DigestMethod
             Algorithm="http://w3.org/xmldsig#sha1"/>
         <ds:DigestValue>lNIzVMrp8CwTE=</></></>
     <ds:SignatureValue>
       GeMp7LS...vnjn8=</></>

Decoding would produce the data structure in Fig-<<see: fig:decode-data>>. You
should also look at c/zx-sa-data.h to see the structs involved in this
example.

<<dot: decode-data: Typical data structure produced by decode.

// This graph crashes dot 1.12, but works in dot 2.8

size="11.0,6.0"
margin=0
rankdir=LR

{ rank=same; siginfo; sigval; }
{ rank=same; canonmeth; sigmeth; ref; }
//{ rank=same; canonmeth; sigmeth; ref; digmeth; digval; }
//{ rank=same; xforms; xform_env; xform_c14n; }
//{ rank=same; xform_env; xform_c14n; digmeth; digval; }
{ rank=same; xforms; digmeth; digval; }
{ rank=same; xform_c14n; xform_env; }

sig [shape=record,label="zx_ds_Signature_s|{|{<f_kids>gg.kids|<f_siginfo>SignedInfo|<f_sigval>SignatureValue|KeyInfo (0)|Object (0)|Id (0)}}"];
siginfo [shape=record,label="zx_ds_SignedInfo_s|{|{<f_kids>gg.kids|<f_wo>gg.g.wo|<f_canonmeth>CanonicalizationMethod|<f_sigmeth>SignatureMethod|<f_ref>Reference|Id (0)}}"];

canonmeth [shape=record,label="zx_ds_CanonicalizationMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xml-exc-c14n#\"}}"];

sigmeth [shape=record,label="zx_ds_SignatureMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xmldsig#rsa-sha1\"}}"];

ref [shape=record,label="zx_ds_Reference_s|{|{<f_kids>gg.kids|gg.g.wo (0)|<f_xforms>Transforms|<f_digmeth>DigestMethod|<f_digval>DigestValue|Id (0)|Type (0)|URI\n\"#RrcrNwFIw6n\"}}"];

xforms [shape=record,label="zx_ds_Transforms_s|{|{<f_kids>gg.kids|<f_wo>gg.g.wo|gg.g.n (0)|<f_xform>Transform}}"];

xform_c14n [shape=record,label="zx_ds_Transform_s|{|{<f_wo>gg.g.wo|gg.g.n (0)|XPath (0)|<f_c14n_algo>Algorithm\n\"http://w3.org/xml-exc-c14n#\"}}"];

xform_env [shape=record,label="zx_ds_Transform_s|{|{gg.g.wo (0)|<f_n>gg.g.n|XPath (0)|Algorithm\n\"http://w3.org/xmldsig#env-sig\"}}"];

xforms:f_xform -> xform_env
xform_env:f_n -> xform_c14n

digmeth [shape=record,label="zx_ds_DigestMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xmldsig#sha1\"}}"];
digval [shape=record,label="zx_elem_s|{|{gg.g.wo (0)|content\n\"lNIzVMrp8CwTE=\"}}"];

sigval [shape=record,label="zx_ds_SignatureValue_s|{|{gg.g.wo (0)|gg.content\n\"GeMp7LS...vnjn8=\"|Id (0)}}"];

sig:f_siginfo -> siginfo
sig:f_sigval  -> sigval

siginfo:f_canonmeth -> canonmeth
siginfo:f_sigmeth -> sigmeth
siginfo:f_ref -> ref

ref:f_xforms -> xforms
ref:f_digmeth -> digmeth
ref:f_digval -> digval

sig:f_kids ->siginfo [weight=0,style=dashed,color=red]

siginfo:f_wo ->sigval [weight=0,style=dashed,color=red]
siginfo:f_kids -> canonmeth [weight=0,style=dashed,color=red]
canonmeth:f_wo -> sigmeth [weight=0,style=dashed,color=red]
sigmeth:f_wo -> ref [weight=0,style=dashed,color=red]

ref:f_kids -> xforms [weight=0,style=dashed,color=red]
xforms:f_wo -> digmeth [weight=0,style=dashed,color=red]
digmeth:f_wo -> digval [weight=0,style=dashed,color=red]

xforms:f_kids -> xform_c14n [weight=0,style=dashed,color=red]
xform_c14n:f_wo -> xform_env [weight=0,style=dashed,color=red]

>>

There are two pointer systems at play here. The black solid arrows
depict the logical structure of the XML document. For each child
element there is a struct field that simply points to the child. If
there are multiple occurrences of the child, as in
~sig->SignedInfo->Reference->Transforms->Transform~, the children are
kept in a linked list connected by gg.g.n (next) fields.<<footnote:
This linked list may be in inverted order depending on the phase of
the moon and position of the trams in Helsinki. Until implementation
matures, its better not to depend on the ordering.>>

The +wide order+ structure, depicted by red dashed arrows, is
maintained using gg.kids and gg.g.wo fields. For example
~sig->SignedInfo->Reference->Transforms~ keeps its kids, the
~zx_ds_Transform~ objects, in the original order hanging from the kids
and linked with the wo field. As can be seen, the order kept with wo
fields can be different than the one kept using n (next) fields.
What's more, the kids list can contain dissimilar objects, witness
~sig->SignedInfo->Reference->gg.kids~. The wire order representation
is only captured when decoding the document and is mainly useful for
correctly canonicalizing the document for signature verification. If
you are building a data structure in your own program, you typically
will not set the gg.kids and gg.g.wo fields.

In the diagram, the objects of type ~zx_str~ were collapsed to
double quoted strings. Superfluous gg.kids, gg.g.wo, and gg.g.n fields
were omitted: they exist in all structures, but are not shown when
they are ~NULL~. The ~NULL~ is depicted as zero (0).<<footnote: All
this gg.g business is just C's way of referencing the fields of a
common base type of element objects.>>


<<notacountry: so wo>>

12.1.1 Handling Namespaces
~~~~~~~~~~~~~~~~~~~~~~~~~~

An annoying feature of XML documents is that they have variable
namespace prefixes. The namespace prefix for the unqualified elements
is taken to be the one specified in target() directive of the .sg
input. Name of an element in C code is formed by prefixing the element
by the namespace prefix and an underscore.

Attributes will only have namespace prefix if such was expressly
specified in .sg input.

When decoding, the actual namespace prefixes are recorded. The wire
order encoder knows to use these recorded prefixes so that accurate
canonicalization for XMLDSIG can be produced.

If the message on wire uses wrong namespaces, the wrong ones are
remembered so that canonicalization for signature validation will work
irrespective. The ability to accept wrong namespaces only works as
long as there is no ambiguity as to which tag was meant - there are
some tags that need namespace information to distinguish. If you hit
one of these then either you get lucky and the one that is arbitrarily
picked by the decoder happens to be the correct one, or you are stuck
with no easy way to make it right. Of course the XML document was
wrong to start with so theoretically this is not a concern. Generally
the more schemata that are simultaneously generated to one package, the
greater the risk of collisions between tags.

The schema order encoder always uses the prefixes defined
using target() directives in .sg files. The runtime notion of
namespaces is handled by ~ns_tab~ field of the decoding and encoding
context.  It is initialized to contain all namespaces known by virtue
of .sg declarations.  The runtime assigned prefixes are held in a
linked list hanging from <<tt: n>> (next) field of ~struct
zx_ns_s~. (*** more work needed here)

The code generation creates a file, such as c/zx-ns.c, which contains
initialization for the table. The main program should point the ~ns_tab~
field of context as follows:

  main {
    struct zx_ctx* ctx;
    ...
    ctx->ns_tab = zx_ns_tab;   /* Here zx_ is the chosen prefix */
  }

Consider the following evil contortion

  <e:E xmlns:e="uri">
    <h:H xmlns:h="uri"/>
    <b:B xmlns:b="uri">
      <e:C xmlns:e="uri"/>
      <e:D xmlns:e="iru">
        <e:F xmlns:e="uri"/></></></>

Assuming the ~ns_tab~ assigns prefix <<tt: y>> to the namespace
URI, we would have following data structure as a result of a decode

<<dot: ns-data,,: Decode of XML and resulting namespace structures.
margin=0
//rankdir=LR

{ rank=same; ns_tab; e; h; b; }
{ rank=same; H; B; }
{ rank=same; C; D; }

ns_tab [shape=record,label="{ns_tab|{y|uri|<uri_n>}|{z|iru|<iru_n>}}"]

e [shape=record,label="e|uri|<n>"]
h [shape=record,label="h|uri|<n>"]
b [shape=record,label="b|uri|0"]
i [shape=record,label="e|iru|0"]

ns_tab:uri_n -> e
ns_tab:iru_n -> i
e:n -> h
h:n -> b

E -> H [style=bold]
E -> B [style=bold]
B -> C [style=bold]
B -> D [style=bold]
D -> F [style=bold]

E -> e [color=red]
H -> h [color=red]
B -> b [color=red]
C -> e [color=red]
D -> i [color=red]
F -> e [color=red]
>>

The red thin arrows indicate how the elements reference the
namespaces. Since none of the elements used the prefix originally
specified in the schema grammar target() directive, we ended up
allocating "alias" nodes for the uri. However, since E and C use the
same prefix, they share the alias node. Things get interesting with D:
it redefines the prefix e to mean different namespace URI, "iru", which
happens to be an alias of prefix z.

Later, when wire order canonical encode is done, the red thin arrows
are chased to determine the namespaces. However, we need to keep a
separate "seen" stack to track whether parent has already declared the
prefix and URI. E would declare xmlns:e="uri", but C would not because
it had already been "seen". However, F would have to declare it again
because the xmlns:e="iru" in D masks the declaration. The ~zx_ctx~
structure is used to track the namespaces and "seen" status
through out decoders and encoders.

<<dot: seen-data,,: Seen data structure (blue dotted and green dashed arrows) in the end of decoding F. S=seen, SN=seen_n.
margin=0
//rankdir=LR

{ rank=same; ns_tab; ee; e; h; b; }
{ rank=same; H; B; }
{ rank=same; C; D; }

ns_tab [shape=record,label="{ns_tab|{P|URI|S|SN|N}|{y|uri|0|0|<uri_n>}|{z|iru|0|0|<iru_n>}}"]

e [shape=record,label="e|uri|0|0|<n>"]
ee [shape=record,label="e|uri|<s>|0|<n>"]
h [shape=record,label="h|uri|0|<sn>|<n>"]
b [shape=record,label="b|uri|0|<sn>|0"]
i [shape=record,label="e|iru|<s>|0|0"]

ctx [shape=record,label="{ctx|{|{<ns>ns_tab|<sn>seen_n}}}"]

ns_tab:uri_n -> ee
ns_tab:iru_n -> i
ee:n -> e
e:n -> h
h:n -> b

E -> H [style=bold]
E -> B [style=bold]
B -> C [style=bold]
B -> D [style=bold]
D -> F [style=bold]

E -> e [color=red]
H -> h [color=red]
B -> b [color=red]
C -> e [color=red]
D -> i [color=red]
F -> ee [color=red]

ns_tab -> ctx:ns [arrowhead=none,arrowtail=normal]
b -> ctx:sn [color=blue,style=dotted,arrowhead=none,arrowtail=normal]
b:sn -> h [color=blue,style=dotted]
h:sn -> ee [color=blue,style=dotted]
ee:s -> i [color=green,style=dashed]
i:s -> e [color=green,style=dashed]
>>

Here we can see how the ~seen_n~ list, represented by the blue dotted
arrows, was built: at the head of the list, ~ctx->seen_n~, is the last
seen prefix, namely b (because, although the meaning of e at F was
different, e as a prefix had already been seen earlier at E), followed
by other prefixes in inverse order of first occurrence.<<footnote: This
is a mere artifact of implementation: it's cheapest to add to the head
of the list. This may change in future.>> The green dashed arrows from
e:uri to e:iru and then on to second e:uri reflect the fact that e:uri
(second) was put to the list first (when we were at E), but later, at
D, a different meaning, iru, was given to prefix e. Finally at F we
give again a different meaning for e, thus pushing to the "seen stack"
another node. Although e at E and at F have namespace URI, "uri", we are
not able to use the same node because we need to keep the stack order.
Thus we are forced to allocate two identical nodes.

12.1.2 Handling any and anyAttribute
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since our aim is to be lax in what we accept, every element can handle
unexpected additional attributes as well as unexpected elements. Thus
whether the schema specifies any or anyAttribute or not, we handle
everything as if they were there. However, when attributes and
elements are received out side of their expected context, they are
simply treated as strings with string names. This is true even for
those attributes and elements that would be recognizable in their
proper context.

The any extension points, as well as some bookkeeping data
are hidden inside ~ZX_ELEM_EXT~ macro. If you tinker with
this macro, be sure you know what you are doing. If you want
to add your own specific fields to all structs, redefining
~ZX_ELEM_EXT~ may be appropriate, but if you want to add more
fields only to some specific structures, you can define
a macro of form

  TPF_EEE_EXT

and put in it whatever fields you want. These fields will be
initialized to zero when the structure is created, but are not touched
in any other way by the generated code. In particular, if some of your
fields are pointers, it will be your responsibility to free them. The
standard free functions will not understand to free them. See the data
structure walking functions, below for one way to accomplish this.

12.1.3 Root data structure
~~~~~~~~~~~~~~~~~~~~~~~~~~

The root data structure

  struct zx_root_s;

is a special structure that has a field for every top level
recognizable element.

12.1.4 Per element data structures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*** TBW

12.1.5 Memory Allocation
~~~~~~~~~~~~~~~~~~~~~~~~

After decoding all string data points directly into the input buffer,
i.e. strings are NOT copied. Be sure to not free the input buffer
until you are done processing the data structure. If you need to take
a copy of the strings, you will need to walk the data structure as a
post processing step and do your copies. This can be done using

  void TPF_dup_strs_len_NS_EEE(struct zx_dec_ctx* c, struct TPF_NS_EEE_s* x);

The structures are allocated via ZX_ZALLOC() macro, which
by default calls zx_zalloc() function, which in turn
uses system malloc(3). However, you can redefine the
macro to use whatever other allocation scheme you desire.

The generated libraries never free(3) memory. In many programming
patterns, this is actually desirable: for example a CGI program can
count on dying - the process exit(2) will free all the memory.

If you need to free(3) the data structure, you will need to walk it
using

  void TPF_free_len_NS_EEE(struct zx_dec_ctx* c,
                           struct TPF_NS_EEE_s* x,
                           int free_strings);
  void zx_free_any(struct zx_dec_ctx* c,
                   struct zx_note_s* n,
                   int free_strs);

The zx_free_any() works by having a gigantic switch statement that calls
the appropriate specific free function.

You can deep clone the data structure with

  void TPF_deep_clone_NS_EEE(struct zx_dec_ctx* c,
                             struct TPF_NS_EEE_s* x,
                             int dup_strings);
  struct zx_note_s* zx_clone_any(struct zx_dec_ctx* c,
                                 struct zx_note_s* n,
                                 int dup_strs);

The zx_clone_any() works by having a gigantic switch statement that calls
the appropriate specific free function.

12.2 Decoder as Recursive Descent Parser
----------------------------------------

The entry point to the decoder is

  struct zx_root_s* zx_DEC_root(struct zx_dec_ctx* c,
                                struct zx_ns_s* dummy,
                                int n_decode);

The decoding context holds pointer to the raw data and must be
initialized prior to calling the decoder. The third argument specifies
how many recognized elements are decoded before returning. Usually you
would specify 1 to consume one top level element from the
stream.<<footnote: The second argument, the dummy namespace, is
meaningless for root node, but makes sense for element decoders. For
root you can simply supply 0 (NULL).>>

The returned data structure, ~struct zx_root_s~, contains
one pointer for each type of top level element that can
be recognized. The ~tok~ field of the returned value
identifies the last top level element recognized and can
be used to dispatch to correct request handler:

  zx_prepare_dec_ctx(c, TPF_ns_tab, start_ptr, end_ptr);
  struct TPF_root_s* x = TPF_DEC_root(c, 0, 1);
  switch (x->gg.g.tok) {
  case TPF_NS_EEE_ELEM: return process_EEE_req(x->NN_EEE);
  }

When processing responses, it is generally already known
which type of response you are expecting, so you can simply
check for NULLness of the respective pointer in the returned
data structure.

Internally zx_DEC_root() works much the same way: it scans
a beginning of an element from the stream, looks up the token
number corresponding to the element name, and switches on
that, calling element specific decoder functions (see next
section) to do the detailed processing.

In the above code fragment, you should note the call to
zx_prepare_dec_ctx() which initializes the decoder machinery.
It takes +ns_tab+ argument, which specifies which namespaces
will be recognized. This table MUST match the TPF_DEC_root()
function you call (i.e. both must have been generated as
part of the same xsd2sg.pl invocation). The other arguments
are the start of the buffer to decode and pointer one past
the end of the buffer to decode.

12.2.1 Element Decoders
~~~~~~~~~~~~~~~~~~~~~~~

For each recognizable element there is a function of form

  struct TPF_NS_EEE_s* zx_DEC_NS_EEE(struct zx_dec_ctx* c);

where TPF is the prefix, NS is the namespace prefix, and
EEE is the element name. For example:

  struct zx_se_Envelope_s* zx_DEC_se_Envelope(struct zx_ctx* c);

These functions work much the same way as the root decoder. You
should consult dec-templ.c for the skeleton of the decoder. Generally
you should not be calling element specific decoders: they
exist so that zx_DEC_root() can call them. They have somewhat
nonintuitive requirements, for example the opening <, the
namespace prefix, and the element name must have already been
scanned from the input stream by the time you call element
specific decoder.

12.2.2 Decoder Extension Points
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The generated code is instrumented with following macros

ZX_ATTR_DEC_EXT(ss):: Extension point called just after decoding known attribute
ZX_XMLNS_DEC_EXT(ss):: Extension point called just after decoding xmlns attribute
ZX_UNKNOWN_ATTR_DEC_EXT(ss):: Extension point called just after decoding unknown attr
ZX_START_DEC_EXT(x):: Extension point called just after decoding element name
    and allocating struct, but before decoding any of the attributes.
ZX_END_DEC_EXT(x):: Extension point called just after decoding the entire element.
ZX_START_BODY_DEC_EXT(x):: Extension point called just after decoding element tag, including attributes, but before decoding the body of the element.
ZX_PI_DEC_EXT(pi):: Extension point called just after decoding processing instruction
ZX_COMMENT_DEC_EXT(comment):: Extension point called just after decoding comment
ZX_CONTENT_DEC(ss):: Extension point called just after decoding string content
ZX_UNKNOWN_ELEM_DEC_EXT(elem):: Extension point called just after decoding unknown element

Following macros are available to the extension points

TPF:: Type prefix (as specified by  -p during code generation)
EL_NAME:: Namespaceful element name (NS_EEE)
EL_STRUCT:: Name of the struct that describes the element
EL_NS:: Namespace prefix of the element (as seen in input schema)
EL_TAG:: Name of the element without any namespace qualification.

12.3 Exclusive Canonical Encoder
--------------------------------

The encoder receives a C data structure and generates a gigantic
string containing an XML document corresponding to the data structure
and the input schemata. The XML document conforms to the rules of
exclusive XML canonicalization and hence is useful as input to XMLDSIG.

One encoder is generated for each root node specified at the code
generation. Often these encoders share code for interior nodes.

The encoders allow two pass rendering. You can first use the length
computation method to calculate the amount of storage needed and
then call one of the rendering functions to actually render. Or
if you simply have large enough buffer, you can just render directly.

The encoders take as argument next free position in buffer
and return a char pointer one past the last byte used. Thus
you can discover the length after rendering by subtracting the
pointers. This is guaranteed to result same length as returned
by the length computation method.<<footnote: This is a useful
sanity check. If the two ever disagree, please report a bug.>>
You can also call the next encoder with the return value
of the previous encoder to render back-to-back elements.

The XML namespace and XML attribute handling of the encoders
is novel in that the specified sort is done already at code
generation time, i.e. the renderers are already in the order
that the sort mandates.

For attributes we know the sort order directly from the schema
because [xml-c14n], sec 2.2, p.7, specifies that they
sort first by namespace URI and then by name, bot of which
we know from the schema.

For ~xmlns~ specifications the situation is similarly easy in the
schema order encoder case because we know the namespace prefixes
already at code generation time. However, for the wire order encoder
we actually need a runtime sort because we can not control which
namespace prefixes get used. However, for both cases we can make a
pretty good guess about which namespaces might need to be declared at
any given element: the element's own namespace and namespaces of each
of its attributes. That's all, and it's all known at code generation
time. At runtime we only need to check if the namespace has already
been seen at outer layer.

12.3.1 Length computation
~~~~~~~~~~~~~~~~~~~~~~~~~

Compute length of an element (and its subelements). The XML attributes
and elements are processed in schema order.

  int TPF_LEN_SO_NS_EEE(struct zx_ctx* c,
                        struct TPF_NS_EEE_s* x);

For example:

  int zx_LEN_SO_se_Envelope(struct zx_ctx* c,
                            struct zx_se_Envelope_s* x);

Compute length of an element (and its subelements). The XML namespaces
and elements are processed in wire order.

  int TPF_LEN_WO_NS_EEE(struct zx_ctx* c,
                        struct TPF_NS_EEE_s* x);

For example:

  int zx_LEN_WO_se_Envelope(struct zx_ctx* c,
                            struct zx_se_Envelope_s* x);

12.3.2 Encoding in schema order
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Render an element into string. The XML elements are processed in
schema order. The xmlns declarations and XML attributes are always
sorted per [xml-exc-c14n] rules.<<footnote: The sort is actually done
already at code generation time by xsd2sg.pl.>> This is what you
generally want for rendering new data structure to a string. The wo
pointers are not used.

  char* TPF_ENC_SO_NS_EEE(struct zx_ctx* c,
                          struct TPF_NS_EEE_s* x,
                          char* p);

For example:

  char* zx_ENC_SO_se_Envelope(struct zx_ctx* c,
                              struct zx_se_Envelope_s* x,
                              char* p);

Since it is a very common requirement to allocate correct
sized buffer and then render an element, a helper function
is provided to do this in one step.

  struct zx_str* zx_EASY_ENC_SO_se_Envelope(struct zx_ctx* c,
                                    struct zx_se_Envelope_s* x);

The returned string is allocated from allocation arena described
by ~zx_ctx~.

12.3.3 Encoding in wire order
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Render element into string. The XML elements are
processed in wire order by chasing wo pointers. This is what you want
for validating signatures on other people's XML documents. If the wire
representation was schema invalid, e.g. elements were in wrong order,
the wire representation is still respected, except for xmlns
declarations and XML attributes, which are always sorted, per exc-c14n
rules. For each element a function is generated as follows

  char* TPF_ENC_WO_NS_EEE(struct zx_ctx* c,
                          struct TPF_NS_EEE_s* x,
                          char* p);

For example

  char* zx_ENC_WO_se_Envelope(struct zx_ctx* c,
                              struct zx_se_Envelope_s* x,
                              char* p);

A helper function is also available

  struct zx_str* zx_EASY_ENC_WO_se_Envelope(struct zx_ctx* c,
                                    struct zx_se_Envelope_s* x);

12.4 Signatures (XMLDSIG)
-------------------------

12.4.1 Signature Generation
~~~~~~~~~~~~~~~~~~~~~~~~~~~

*** TBW

12.4.2 Signature Validation
~~~~~~~~~~~~~~~~~~~~~~~~~~~

For signature validation you need to walk the decoded data structure
to locate the signature as well as the references and pass them to
zxsig_validate(). The validation involves wire order exclusive
canonical encoding of the referenced XML blobs, computation of SHA1 or
MD5 checksums over them, and finally computation of SHA1 check sum
over the <SignedInfo> element and validation of the actual
<SignatureValue> against that. The validation involves public key
decryption using the signer's certificate.

A nasty problem in exclusive canonicalization is that the namespaces
that are needed in the blob may actually appear in the containing XML
structures, thus in order to know the correct meaning of a namespace
prefix, we need to perform the +seen+ computation for all elements
outside and above the blob of interest.<<footnote: This is yet another
indication of how botched the XML namespace concept is. Or this could
have been fixed in the exclusive canonicalization spec by not using
namespace prefixes at all.>>

To verify signature, you have to do certain amount of preparatory work
to locate the signature and the data that was signed. Generally what
should be signed will be evident from protocol specifications or from
the security requirements of your application environment. Conversely,
if there is a signature, but it does not reference the appropriate
elements, its worthless and you might as well reject the document
without even verifying the signature.

*Example*

    struct zxsig_ref refs[1];
    cf = zxid_new_conf("/var/zxid/");
    ent = zxid_get_ent_from_file(cf, "YV7HPtu3bfqW3I4W_DZr-_DKMP4.");
    
    refs[0].ref = r->Envelope->Body->ArtifactResolve
                   ->Signature->SignedInfo->Reference;
    refs[0].blob = (struct zx_elem_s*)r->Envelope->Body->ArtifactResolve;
    res = zxsig_validate(cf->ctx, ent->sign_cert,
                         r->Envelope->Body->ArtifactResolve->Signature,
                         1, refs);
    if (res == ZXSIG_OK) {
      D("sig vfy ok %d", res);
    } else {
      ERR("sig vfy failed due to(%d)", res);
    }

This code illustrates

1. You have to determine who signed and provide the entity
   object that corresponds to the signer. Often you
   would determine the entity from <Issuer> element somewhere
   inside the message.

   The entity is used for retrieving the signing certificate.
   Another alternative is that the signature itself contains
   a <KeyInfo> element and you extract the certificate from
   there. You would still need to have a way to know if you
   trust the certificate.

2. You have to prepare the refs array. It contains pairs of
   <SignedInfo><Reference> specifications combined with the
   actual elements that are signed. Generally the URI
   XML attribute of the <Reference> element points to the
   data that was signed. However, it is application dependent
   what type of ID XML attribute the URI actually references
   or the URI could even reference something outside the
   document. It would be way too unreliable for the
   zxsig_validate() to attempt guessing how to locate the
   signed data: therefore we push the responsibility to
   you. Your code will have to walk the data to locate
   all referenced bits and pieces.

   In the above example, locating the one signed bit was
   very easy: the specification says where it is (and this
   location is fixed so there really is no need to check
   the URI either).

   You pass the length of the refs array and the array
   itself as two last arguments to zxsig_validate().

3. You need to locate the <Signature> element in the document
   and pass it as argument to zxsig_validate(). Usually
   a protocol specification will say where the <Signature>
   element is to be found, so locating it is not difficult.

4. The return value will indicate validation status. ZXSIG_OK,
   which has numerical value of 0, indicates success. Other
   nonzero values indicate various kinds of failure.

12.4.3 Certificate Validation and Trust Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Trust models for TLS and signature validation are separate. TLS layer
is handled mainly by libcurl or in case of ClientTLS, by the https web
server (which is not part of zxid).

In signature validation the primary trust mechanism is that entity's
metadata specifies the signing certificate and there is no
Certification Authority check at all.<<footnote: If you develop CA
check, please submit patches to ZXID project.>>
This model works well if you control the admission
to your CoT. However, ZXID ships by default with the
automatic CoT feature turned on, thus anyone can get
added to the CoT and therefore signature with any
certificate they declare is "valid". This hardly
is acceptable for anything involving money.

12.5 Data Accessor Functions
----------------------------

Simple read access to data should, in C, be done by
simply referencing the fields of the struct, e.g.

  if (!r->EntitiesDescriptor->EntityDescriptor)
      goto bad_md;

*** TBW

12.6 Memory Allocation and Free
-------------------------------

*** TBW

12.7 Walking the data structure
-------------------------------

*** TBW

12.9 Thread Safety
------------------

All generated libraries are designed to be thread safe, provided
that the underlying libc APIs, such as malloc(3) are thread safe.

13 ID-WSF Features of ZXID
==========================

13.1 EPR Cache
--------------

Calling an ID Web Service requires an Endpoint Reference (EPR) that
indicates not only the URL of the web service, but also the security
mechanism and any tokens or credentials required by the security
mechanism, collectively called the +metadata+ of the EPR.<<footnote:
While the role of the EPR metadata is broadly similar to SAML
metadata, the two should not be confused.>>

An EPR can be obtained from two sources

1. From the SSO assertion containing an attribute statement that
   has the EPR. This is called the +bootstrap+ method.

2. By calling the discovery service.

Either way, the EPRs are cached in the SSO session as files
with paths like

  /var/zxid/ses/SESID/SVCTYPE,SHA1

where +SESID+ is the SSO session ID (safe base64 of a random number).
+SVCTYPE+ and +SHA1+ form a unique file name inside the session directory.
+SVCTYPE+ component allows easy identification of the EPRs that are
relevant for calling a given service. The +SHA1+ component is a SHA1 hash
over the canonical XML representation of the EPR.

When bootstrap EPRs are present in the SSO assertion, they are
automatically extracted to the EPR cache (internally
zxid_snarf_eprs_from_ses() is called). Generally this will yield at
least bootstrap EPR for discovery service. Later, when
calling web services, discovery is automatically performed
as needed and the results are automatically populated to the
EPR cache (internally zxid_cache_epr() is called).

When calling higher level WSC APIs, the discovery will happen
automatically, but if you work in terms of lower level APIs,
you can obtain an EPR by calling zxid_get_epr() which will
first consult the cache, and if there is a miss, then try discovering
the EPR using discovery service EPR, if any (usually one was obtained
as bootstrap during the SSO).

13.2 Low Level WSC API
----------------------

Typical code for calling a web service at low level

  01 #include <zx/errmac.h>
  02 #include <zx/zxid.h>
  03 #include <zx/zxidconf.h>
  04 #include <zx/saml2.h>
  05 #include <zx/wsf.h>
  06 #include <zx/c/zx-ns.h>
  07
  08 struct zx_e_Envelope_s* env;
  09 struct zx_a_EndpointReference_s* epr;
  10 epr = zxid_get_epr(cf, ses, zx_xmlns_dap, 1);
  11 if (epr) {
  12   env = zx_NEW_e_Envelope(cf->ctx);
  13   env->Header = zx_NEW_e_Header(cf->ctx);
  14   env->Body = zx_NEW_e_Body(cf->ctx);
  15   env->Body->Query = zxid_mk_dap_query(cf, ...); /* See ID-DAP inteface */
  16   env = zxid_wsc_call(cf, ses, epr, env);        /* The beef */
  17   if (env->dap_QueryResponse)
  18     D("Result is LDIF(%.*s)",
  19        env->Body->dap_QueryResponse->Data->LDIF->gg.content->len,
  20        env->Body->dap_QueryResponse->Data->LDIF->gg.content->s);
  21 }

1. On line 10 zxid_get_epr() is used with following arguments
   cf:: Configuration object
   ses:: Session object (generallly obtained from SSO)
   zx_xmlns_dap:: The +service type+ of the service that is
       to be called. Generally this is same as the namespace
       URI. The include <zx/c/zx-ns.h> contains macros,
       such as ~zx_xmlns_dap~, for the namespaces supported
       by zxid. Alternatively you could simply supply the string.
   1:: The last argument ("1", one) is an iterator index that, in case of
       multiple EPRs allows you to pick which one to use.
       Most common usage is to simply supply 1 which picks
       the first one.<<footnote: However, the ordering of the
       eprs in not currently well defined and may change
       from one version of ZXID to another.>>

13.3 ID-DAP Interface
---------------------

The ID-DAP search filters and data model are very similar to
LDAP, see [RFC2251] for further explanation.

13.3.1 Short example of using low level API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  env->Body->dap_Query
    = zxid_mk_dap_query(cf,
          0,   /* No tests */
          zxid_mk_dap_query_item(cf,
                                 zxid_mk_dap_select(cf,
                                                    0,  /* DN from ID-WSF */
                                                    "objecttype=svcprofile",
                                                    0,  /* all attributes */
                                                    1,  /* chase symlinks */
                                                    ZXID_DAP_SCOPE_SUBTREE,
                                                    0,  /* no size limit */
                                                    0,  /* no time limit */
                                                    0), /* return data */
                                 0,  /* regular data entries */
                                 0,  /* No predefined operation */
                                 0,  /* No sorting. */
                                 0,  /* No changed since specification. */
                                 0,  /* Do not include LDAP common attributes. */
                                 0,  /* Start from first result (offset == 0) */
                                 0,  /* Return all results (count == 0) */
                                 0,  /* Do not request snapshot */
                                 0,  /* Do not refer to snapshot */
                                 0), /* No contingent item ID reference */
          0);  /* No subscriptions */

As can be seen, it is common to specify nearly all arguments as 0,
relying on default values. Thus the code typically appears
without comments as

  env->Body->dap_Query
    = zxid_mk_dap_query(cf, 0,
          zxid_mk_dap_query_item(cf,
              zxid_mk_dap_select(cf, 0, "objecttype=svcprofile",
                                 0, 1, ZXID_DAP_SCOPE_SUBTREE, 0, 0, 0),
              0, 0, 0, 0, 0, 0, 0, 0, 0, 0),  /* 10 zeroes here */
          0);  /* No subscriptions */


13.3.2 Fully winded example of using low level API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As a query can also have test and subscription clauses, the
fully winded example becomes quite onerous. Of course for many
daily tasks you will not need all the frills, or you can
just use the simple API instead.

  env->Body->dap_Query
    = zxid_mk_dap_query(cf,
          zxid_mk_dap_test_item(cf,
                                zxid_mk_dap_testop(cf,
                                                   0,  /* DN from ID-WSF */
                                                   "objecttype=svcprofile",
                                                   0,  /* all attributes */
                                                   1,  /* chase symlinks */
                                                   ZXID_DAP_SCOPE_SUBTREE,
                                                   0,  /* no size limit */
                                                   0,  /* no time limit */
                                                   0), /* return data */
                                0,   /* regular data entries */
                                0),  /* No predefined operation */
          zxid_mk_dap_query_item(cf,
                                 zxid_mk_dap_select(cf,
                                                    0,  /* DN from ID-WSF */
                                                    "objecttype=svcprofile",
                                                    0,  /* all attributes */
                                                    1,  /* chase symlinks */
                                                    ZXID_DAP_SCOPE_SUBTREE,
                                                    0,  /* no size limit */
                                                    0,  /* no time limit */
                                                    0), /* return data */
                                 0,  /* regular data entries */
                                 0,  /* No predefined operation */
                                 0,  /* No sorting. */
                                 0,  /* No changed since specification. */
                                 0,  /* Do not include LDAP common attributes. */
                                 0,  /* Start from first result (offset == 0) */
                                 0,  /* Return all results (count == 0) */
                                 0,  /* Do not request snapshot */
                                 0,  /* Do not refer to snapshot */
                                 0), /* No contingent item ID reference */
          zxid_mk_dap_subscription(cf,
              "subsID",
              0,  /* No item ID reference */
              zxid_mk_dap_resquery(cf,
                                   zxid_mk_dap_select(cf,
                                                      0,  /* DN from ID-WSF */
                                                      "objecttype=svcprofile",
                                                      0,  /* all attributes */
                                                      1,  /* chase symlinks */
                                                      ZXID_DAP_SCOPE_SUBTREE,
                                                      0,  /* no size limit */
                                                      0,  /* no time limit */
                                                      0), /* return data */
                                   0,  /* regular data entries */
                                   0,  /* No predefined operation */
                                   0,  /* No sorting. */
                                   0,  /* No changed since specification. */
                                   0,  /* Do not include LDAP common attributes. */
                                   0), /* No contingent item ID reference */
              0, /* No notification aggregation spec. */
              0, /* No notification trigger spec. */
              0, /* Subscription starts immediately. */
              0, /* Subscription never expires. */
              1, /* Include changed data in the notifications. */
              0, /* Use notification reference for administrative notifications. */
              "http://host/notif_sink")
          );

13.3.3 zxid_mk_dap_query()
~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_Query_s* zxid_mk_dap_query(struct zxid_conf* cf,
                                           struct zx_dap_TestItem_s* tis,
                                           struct zx_dap_QueryItem_s* qis,
                                           struct zx_dap_Subscription_s* subs);

13.3.4 zxid_mk_dap_query_item()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_QueryItem_s* zxid_mk_dap_query_item(struct zxid_conf* cf,
                                                    struct zx_dap_Select_s* sel,
                                                    char* objtype,
                                                    char* predef,
                                                    char* sort,
                                                    char* changed_since,
                                                    int incl_common_attrs,
                                                    int offset,
                                                    int count,
                                                    char* setreq,
                                                    char* setid,
                                                    char* contingent_itemidref);

sel:: Selection expression, see zxid_mk_dap_select().
objtype:: Either "entry", or "_Subsciption". The former is used for
    searching and manipulating the normal attribute data while the
    latter is used for manipulating subscription objects. Specifying
    NULL selects "entry".
predef:: Predefined serverside operation identifier. This
    is server dependent, but you can think of it as a stored procedure.
    If you pass predef, it is common to leave sel and objtype
    as NULL and vice versa, i.e. you should pass NULL as
    predef unless you know your server to expect otherwise.
sort:: String specifying sorting of the result set. See ID-DAP
    specification for further details. N.B. Not all servers
    support sorting. Pass NULL if you do not need sorting.
changed_since:: LDAP date time string specifying that only
    entries that have changed since specified moment
    should be returned.
incl_common_attrs:: If 1 (true), "common" LDAP attributes such
    as modificationtime are included. If you do not need these
    attributes you can save the server some work and also
    some network transmission overhead by not requesting
    these attributes, i.e. pass 0 (false).
offset:: If search matches multiple entries, the zero based
    index of the first entry to return. Pass 0 to get the
    beginning of the result set.
count:: Maximum number of entries to return in the response. 0 means return
    all. +offset+ and +count+ allow pagination through large
    result set. N.B. +count+ is very different from +sizelimit+
    that you may specify in zxid_mk_dap_select(). The latter
    causes the actual backend search operation to abort or return
    partial result set if the result would be too large and
    is unsuitable for pagination.
setreq:: Request creation of a result set snapshot for pagination.
    N.B. It is possible to paginate through large result
    set even without creating a snapshot, but the results
    may be inconsistent because the underlying data may change
    between queries that paginate through it. Creating
    a snapshot avoids this problem. Whether pagination with
    or without snapshot is cheaper depends on the backend
    implementation. +setreq+ is specified on the first query
    that referes to the snapshot. The subsequent queries
    referring to the same snapshot will specify +setid+. The
    two are mutually exclusive: if +setid+ is specified
    the +setreq+ must be NULL.
setid:: If you are paginating through a result set snapshot
    created using +setreq+, then you must specify +setid+
    in subsequent queries that refer to same snapshot.
    When specifying +setreq+, you muse leave +setid+ as NULL.
contingent_itemidref:: A query item can be made contingent
    on a test item, i.e. the query will only be made if the
    test succeeded. If you want this, you must pass the
    item ID of the test item here. Passing NULL means that
    no such dependency exists. N.B. The server side
    implementation of the tests may actually require a query
    to be made anyway.

13.3.5 zxid_mk_dap_select()
~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_Select_s* zxid_mk_dap_select(struct zxid_conf* cf,
                                             char* dn,
                                             char* filter,
                                             char* attributes,
                                             int deref_aliases,
                                             int scope,
                                             int sizelimit,
                                             int timelimit,
                                             int typesonly);

dn:: Distinguished or relative distinguished name. Since ID-DAP usually
    uses the identity conveyed using ID-WSF headers to determine
    the distinguished name, it is common to pass simply a NULL.
filter:: LDAP filter to apply. NULL if none.
attributes:: List of attributes to return. NULL means return all attributes.
deref_aliases:: Boolean: whether the server should chase any
    "symlinks", i.e. an entry may appear at some location as
    an alias that is just a pointer to the real location of the entry.
    This is usually what you want so pass 1 (true).
scope:: The scope of the ID-DAP search.
    ZXID_DAP_SCOPE_BASE (0):: Only what is pointed to by DN, e.g. one
        entry. The default.
    ZXID_DAP_SCOPE_SINGLE  (1):: Single level of directory right under DN.
    ZXID_DAP_SCOPE_SUBTREE (2):: Full subtree search under the DN.
sizelimit:: Maximum number of entries to return. 0 means no limit. This
    is intended to stop the server from accidentally performing
    expensive queries. N.B. +sizelimit+ is different from +count+,
    see zxid_mk_dap_query_item(), the latter is meant for pagination
    of a large result set without aborting it.
timelimit:: Maximum number of seconds to spend in the search. 0 means no limit.
typesonly:: If true, only attribute names are returned, without their
    values. This allows an existence test to be performed without
    passing the values over the network. Usually you want the
    values so you would pass 0 (false).

13.3.6 zxid_mk_dap_test_item()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_TestItem_s* zxid_mk_dap_test_item(struct zxid_conf* cf,
                                                  struct zx_dap_TestOp_s* tstop,
                                                  char* objtype,
                                                  char* predef);

tstop:: See zxid_mk_dap_testop().
objtype:: See +objtype+ in zxid_mk_dap_query_item().
predef:: See +predef+ in zxid_mk_dap_query_item().

13.3.7 zxid_mk_dap_testop()
~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_TestOp_s* zxid_mk_dap_testop(struct zxid_conf* cf,
                                             char* dn,
                                             char* filter,
                                             char* attributes,
                                             int deref_aliases,
                                             int scope,
                                             int sizelimit,
                                             int timelimit,
                                             int typesonly);

See description of zxid_mk_dap_select(). For ID-DAP protocol the
Select and TestOp are defined to be the same. However, this need not be
the case for Data Services Template (DST) based services in
general. Hence, the data types for Select and TestOp are different
(although very similar) and two separate constructors are needed.

13.3.8 zxid_mk_dap_subscription()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_Subscription_s* zxid_mk_dap_subscription(struct zxid_conf* cf,
                                                         char* subsID,
                                                         char* itemidref,
                                                         struct zx_dap_ResultQuery_s* rq,
                                                         char* aggreg,
                                                         char* trig,
                                                         char* starts,
                                                         char* expires,
                                                         int incl_data,
                                                         char* admin_notif,
                                                         char* notify_ref);

subsID:: Subscription ID
itemidref:: When subscribing to data described by a query item,
    create item, or modify item, the reference to the relevant item.
    NULL if no such item exists, in which case +rq+ is usually specified.
rq:: Result query that identifies the data of interest for the
    subscription. See zxid_mk_dap_resquery(). Pass NULL if the
    data is identified otherwise, e.g. via +itemidref+.
aggreg:: Notification aggregation mode. Implementation dependent. Pass NULL.
    Notification aggregation is an optimization where some notification
    may be delayed a little so that it can be sent more optimally
    in same message with other notifications that may happen a little
    later. This functionality need not be supported by the backend
    implementations.
trig:: Implementation dependent notification triggers. Pass NULL.
starts:: Start date time of the subscription. NULL means subscription
    starts immediately.
expires:: End data time of the subscription. NULL means the subscription
    will not expire.
incl_data:: If 1 (true), the notifications resulting from the
    subscription will contain the changed data (push model). If
    0 (false), the notification will just say that something changed,
    but interested party will need to perform a separate query
    to retrieve the data (pull model).
admin_notif:: Administrative notification address. If NULL, the
    +notify_ref+ will be used for administrative notifications
    as well.
notify_ref:: Notification address.

13.3.9 zxid_mk_dap_resquery()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  struct zx_dap_ResultQuery_s* zxid_mk_dap_resquery(struct zxid_conf* cf,
                                                    struct zx_dap_Select_s* sel,
                                                    char* objtype,
                                                    char* predef,
                                                    char* sort,
                                                    int incl_common_attr,
                                                    char* changed_since,
                                                    char* contingent_itemidref);

See description of zxid_mk_dap_query_item().


13.4 ID Messaging Interface
---------------------------

13.5 ID Geo Location Interface
------------------------------

13.6 Contact Book Interface
---------------------------

13.7 People Service Interface
-----------------------------

13.8 Interface to Conor's Demo Media Service
--------------------------------------------

13.9 ID-SIS Data Service for HR-XML
-----------------------------------

HR-XML defines a XML document format that can be used for data
interchange in the Human Resources world. The essence of HR-XML is to
represent CV (Curriculum Vitae, a.k.a. Resume) in structured form.

At the moment (June 2007) HR-XML does not define any standard way to
pass these documents around. In CV 2007 conference organized by EIfE-L
in Paris, an idea was canvassed: use Liberty Data Service to exchange
HR-XML documents. I tried to implement such service during the
conference, but only got it working at the air port after the
conference.

This service is a good example of how to apply ZXID to implement
your own services.

*Files*

  sg/hr-xml-sampo.sg   - A slightly modified version of HR-XML schema
  sg/id-hrxml.sg       - Liberty DST 2.1 based data service for HR-XML
  zxidhrxmlwsc.c       - Web Services Client for HR-XML
  zxidhrxmlwsp.c       - Web Services Provider for HR-XML

*Running demo*

1. Set up your /etc/hosts. At least you need sp1.zxidsp.org
   and you may also need your IdPs domain name (e.g. idp.symdemo.com).

2. Start your IdP and DS (not supplied with ZXID)

    cd /opt/SYMfiam/std
    conf/test-idp3/start.sh start log

   Next you need to cause the id-hr-xml services to be registered
   in the discovery service. This depends on product. (*** fix)

   Then you need to create an association for id-hr-xml service
   for some test user.

3. Start the web services client

    mini_httpd -p 8443 -c 'zxid*' -S -E zxid.pem -l tmp/mini.stderr &
    tail -f tmp/zxid.stderr&

4. Start the web services provider

    mini_httpd -p 8444 -c 'zxid*' -S -E zxid.pem -l tmp/mini2.stderr &
    tail -f tmp/zxid2.stderr&

5. Start browsing from

    https://sp1.zxidsp.org:8443/zxidhrxmlwsc?o=E

6. Login using the test user that has the association

7. Paste <Candidate> element in HR-XML Data form field and click Create. This
   creates a Candidate record in the WSP.

8. Click Query. This queries the WSP for the record we saved in previous step.
   You should see the record appear in the HR-XML Response field.

http://s-idp.liberty-iop.org:8881/N

14 Integration of Other Libraries with ZXID
===========================================

14.1 Conor Cahill's C++ Library for ID-WSF
------------------------------------------

Conor P. Cahill, of AOL and Intel fame, has developed and maintains a
C++ library for ID-WSF 2.0 Web Service Client functionality for
selected application protocols, including the ID-WSF 2.0 Discovery and
some application protcols. Conor also provides a server side package
that implements the corresponding WSP roles in Java. These libraries
are valuable resources and come with extensive test suites - in fact,
passing Conor's test suites has become the gold standard for validity
and interoperability of any ID-WSF implmentations (this is not to
detract from formal IOP events and the Liberty certification program,
but passing Conor's test suite is a good predictor of getting
certified).

*Install Recipe*

Conor's libraries have certain dependencies. Following is my best understanding
of how to get them installed.<<footnote: As of May 2007, Conor's packages
explode in the current working directory. I recommend creating a wrapper
directory first. Also, the client and server functionality can not be
unpacked in same directory without creating conflict and overwriting some files.>>

  mkdir conor
  cd conor
  tar xvf /t/LibertyIDWSFServices-v0.8.2.tgz 
  cd ..
  mkdir conor-cli
  cd conor-cli/
  tar xvf /t/LibertyClientToolkit-v1.0.1.tgz 

14.2 Pat Patterson's php module
-------------------------------

Pat Patterson of Sun distributes a pure PHP module (not to be confused
with Sun's OpenSSO open source effort, with which Pat has some
contact) that implements some aspects of SAML 2.0. As of May 2007, his
library provides functionality that, by and large, parallels that of the
php_zxid module. A major advatage of his modle is that it does not have
C shared library dependency, but beware that he still depends on XML
parsing and popular crypto libraries (openssl) to be available. These
assumptions are not onerous, but you should be aware of them in case
your system differs from main stream deployments.

Overall, Pat's PHP implementation, as of May 2007, is still lacking
in metadata generation and loading (it does not implement Auto-CoT
or Well Known Location) and has some rough edges around less frequently
used parts of the SAML specification. No doubt matters will improve
over the time.

Pat's library handles only SSO and not ID Web Services. It would be
possible to extract the discovery bootstrap from SSO using his library
after which you can use ZXID WSC API to actually call the services.

14.3 Sun OpenSSO
----------------

Sun Microsystems distributes an open source implementation of SAML 2.0.
Their implementation is of primary interest as it provides a freely available
IdP implementation (as of May 2007 IMNSHO the ZXID SP interface is
superior to the OpenSSO SP - and since both implement an open standard,
you can mix ZXID SP with OpenSSO IdP).

Thus, the ZXID to OpenSSO integration reduces to each one acting in its
role using standard wire protocol - SAML 2.0.

15 Creating New Interfaces Using ZXID Methodology
=================================================

The ZXID code generation methodology can be used to create
interfaces to any XML document or protocol that can be
described as a Schema Grammar (which includes any document
that can be expressed as XML Schema - XSD). The general
steps are

1. Convert .xsd file to .sg, or write the .sg directly. For conversion,
   you would typically use a command like

     ~/pd/xsd2sg.pl <foo.xsd >foo.sg

2. Tweak and rationalize the resulting .sg file. In ideal world
   any construct expressible as .xsd should be nicely representable,
   but in practise some work better than others, thus you can create
   a much nicer interface if you invest in some manual tweaking.

   Note that the tweaked .sg still is able to represent the
   same document as the original .xsd described, though
   often the tweaking causes some relaxation.

   Most common tweaks

   a. If the .xsd is written so that the targeted namespace is
      also the default namespace, you should introduce
      a namespace prefix because this is needed during
      code generation to keep different C identifiers
      from clashing with each other. Ideally you
      should coordinate the namespace prefixes globally
      so that even two different projects will not clash.

   b. Where the choice construct is used, indicated
      by pipey symbol (|) in the .sg file, you
      should refactor these into sequences of
      zero-or-one occurrence (?) instances of the alternatives
      of the choice. This is needed because for the foreseeable
      future xsd2sg.pl has a limitation in code generation
      feature. If the choice has maxOccurs="unbounded"
      you should use (*) instead.

   c. xml:lang and other similar attributes may need to
      be factored open to be just of type %xs:string. This
      is a bug in xsd2sg.pl
      
3. "Connect" the schema to bigger framework. Usually this
   means adding your schema grammar to the ZX_SG variable
   in zxid/Makefile and supplying additional -r flags
   in ZX_ROOT variable. This allows your new schema to
   be visible at top level.

   If your schema is meant to extend leafs or interior nodes of
   the parse tree, such as SOAP Body, you would edit
   the SOAP schema to accept your
   new protocol elements in the Body. Or that the generic SOAP
   header can accept your specific header schemata, or that
   the SAML attribute definitions accept your kind of
   attributes - whatever makes sense in your context.

   Alternative to this is to create an entirely new
   monolithic encoder decoder, i.e. instead of extending
   the existing ZXID project to accommodate your new
   protocol, you just start a new project that uses the
   same methodology. You should see how the SAML protocol
   part is separated from the SAML metadata parsing and
   from the WSF parsing in the existing project.

16 ZXID Project
===============

Immediate goal: build a SAML 2.0 SP and ID-WSF 2.0 WSC

Goals of ZXID project include

* SOAP 1.1 support (done)
* SAML 2.0 compliance
  - SP role (done)
  - IdP role
* Liberty ID-FF 1.2 support
  - SP
  - IdP
  - SAML 1.1
* Liberty ID-WSF 1.1 support
  - Discovery bootstrap
  - Discovery WSC
  - ID-DAP WSC
  - ID-DAP WSP
* Liberty ID-WSF 2.0 support
  - Discovery bootstrap (done)
  - Discovery WSC (done)
  - ID-DAP WSC (done)
  - ID-DAP WSP

<<table: ZXID Platform Support
Platform        Native     Cross Compile    Notes
=============== ========== ================ ================================
Linux-ix86      gcc-3.4.6  n/a              Development platform
Solaris 8-sparc gcc-3.4.6  Linux gcc-3.4.6  Fully functional
Windows 2000    -          Linux gcc-3.4.6  Poorly tested
xBSD/Unix       gcc-3.4.6  n/a              C core tested, language bindings not tested
>>

<<table: ZXID Feature and Language Support (version number indicates last testing)
Feature               C     mod_perl mod_php Python Java/Tomcat Apache Shell
===================== ===== ======== ======= ====== =========== ====== =====
Geo Location          Alpha
ID-MM7                Alpha
ID-DAP                Beta
ID-HR-XML             Beta
Contact Book          Alpha
People Service        Alpha
Discovery             Yes
Web Services (ID-WSF) Yes
SSO                   0.17  0.17     0.17    Plan   0.17        Plan   0.17
>>

<<table: ZXID Enabled Application Packages
Application    Language      Notes
============== ============= =====================================================
DokuWiki       PHP           Patch available, in process of submitting to DokuWiki authors
>>

<<ignore: table: ZXID Enabled Application Packages
Application    Language      Notes
============== ============= ==============================
MediaWiki      PHP           Planned
Cognito
zxbug          Perl          Planned
>>

16.1 Project Layout
-------------------

Following directory layout is used by the project. Many of the specified
directories are used by intermediate outputs that are not distributed
in tarball releases, but may or may no be present in CVS checkouts.

  zxid-0.xx
   |
   +-- Net       The Net::SAML perl module (also mod_perl)
   +-- php       PHP / mod_php integration
   +-- zxidjava  The Java JNI interface to ZXID
   +-- servlet   Apache Tomcat integration
   +-- c         C code generated from the Schema Grammar descriptions
   +-- sg        Schema Grammar (.sg) descriptions of protocols
   +-- xsd       XML schema descriptions of protocols (not distributed)
   +-- tex       Temporary files for document generation using PlainDoc (not distributed)
   +-- html      HTML documentation generated using PlainDoc
   +-- review    Publicly released announcements and documents (not distributed)
   +-- t         Test scripts and expected test outputs
   `-- tmp       Temporary files, such as actual test outputs

The Manifest file, which follows, explains each file in more detail.

<<logoutput:
<<Manifest>>
>>

16.2 Protocol Encoders and Decoders
-----------------------------------

The protocol encoders and decoders are generated automatically from
the schema grammar (.sg) descriptions. This ensures accurate protocol
implementation. While the output is strictly schema driven and correct,
the decoders have some provisions to accept some deviations from
strict spec (e.g. out of order elements are tolerated). However,
one should note that XMLDSIG does not tolerate very much deviation,
thus even if decoder accepts a slightly illformed message, it is likely
to fail in signature verification.

There are three outputs from generation

1. Data structures describing the data (xx.h)
2. Encoder that linearizes the data structure to wire protocol (xx-enc.c)
3. Decoder that converts wire protocol byte stream to a data structure (xx-dec.c)

16.3 Standards and Namespaces
-----------------------------

ZXID uses consistently the same namespace prefixes throughout the project. The
generated encoders and decoders support following schemata

<<longtable: ZXID Namespace Convention
Prefix URI                                            Description
====== ============================================== =================================
sa     urn:oasis:names:tc:SAML:2.0:assertion          SAML 2.0
sp     urn:oasis:names:tc:SAML:2.0:protocol
md     urn:oasis:names:tc:SAML:2.0:metadata
ecp    urn:oasis:names:tc:SAML:2.0:profiles:SSO:ecp
paos   urn:liberty:paos:2006-08
sa11   urn:oasis:names:tc:SAML:1.0:assertion          SAML 1.1
sp11   urn:oasis:names:tc:SAML:1.0:protocol
ff12   urn:liberty:iff:2003-08                        ID-FF 1.2
m20    urn:liberty:metadata:2004-12                   v2.0 (almost same as 1.2)
ac     urn:liberty:ac:2004-12                         v2.0 (almost same as 1.2)
b12    urn:liberty:sb:2003-08                         ID-WSF 1.1 SOAP Binding
sec12  urn:liberty:sec:2003-08                        ID-WSF 1.1 Security Mechanisms
di12   urn:liberty:disco:2003-08                      ID-WSF 1.1 Discovery Service
is12   urn:liberty:is:2003-08                         ID-WSF 1.1 Interaction Service
lu     urn:liberty:util:2006-08                       ID-WSF 2.0 Utility Schema
sbf    urn:liberty:sb                                 Framework header
b      urn:liberty:sb:2006-08                         ID-WSF 2.0 SOAP Binding
sec    urn:liberty:security:2006-08                   ID-WSF 2.0 Security Mechanisms
di     urn:liberty:disco:2006-08                      ID-WSF 2.0 Discovery Service
is     urn:liberty:is:2006-08                         ID-WSF 2.0 Interaction Service
dap    urn:liberty:id-sis-dap:2006-08:dst-2.1         ID Directory Access Protocol
dst    urn:liberty:dst:2006-08                        Data Services Template 2.1
subs   urn:liberty:ssos:2006-08                       Subscription and Notification
ps     urn:liberty:ps:2006-08                         People Service
im     urn:liberty:ims:2006-08                        Identity Mapping svc (aka Token Map)
as     urn:liberty:sa:2006-08                         ID-WSF 2.0 Authentication Service
cb     urn:liberty:id-sis-cb:2004-10                  Contact Book Protocol (DST 2.0 based)
cdm    urn:liberty:cb:conceptual-data-model:2004-10   Contact Book Common Data Model
gl     urn:liberty:id-sis-gl:2005-07                  Geolocation Service

mm7
http://www.3gpp.org/ftp/Specs/archive/23_series/23.140/schema/REL-6-MM7-1-4
ID-MM7 (ID-SIS-CSM)
dp     urn:liberty:dp:2006-12                         ID-WSF 2.0 Design Patterns
idp    urn:liberty:idp:2006-12                        ID-WSF 2.0 IdP as web svc
pmm    urn:liberty:pmm:2006-12                        ID-WSF 2.0 Prov Mod Mgr
prov   urn:liberty:prov:2006-12                       ID-WSF 2.0 TM Provisioning
shps   urn:liberty:shps:2006-12                       ID-WSF 2.0 Svc Handling and Proxying
e      http://schemas.xmlsoap.org/soap/envelope/      SOAP 1.1, with SAML and WSF
xa     urn:oasis:names:tc:xacml:2.0:policy:schema:os  XACML 2.0
xac    urn:oasis:names:tc:xacml:2.0:context:schema:os
xasp   urn:oasis:xacml:2.0:saml:protocol:schema:os
xasa   urn:oasis:xacml:2.0:saml:assertion:schema:os

wst
http://docs.oasis-open.org/ws-sx/ws-trust/200512/
WS-Trust 1.3 CD-01
wsp    http://schemas.xmlsoap.org/ws/2004/09/policy

wsc
http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512
WS-Secure Conversation CD-01
ds     http://www.w3.org/2000/09/xmldsig#             XML Signatures
xenc   http://www.w3.org/2001/04/xmlenc#              XML Encryption
exca   http://www.w3.org/2001/10/xml-exc-c14n#        Exclusive Canonicalization
a      http://www.w3.org/2005/08/addressing           WSA 1.0

wsse
http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
WS Security SecExt 1.0

wsu
http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
WS Security Utility 1.0
xml    http://www.w3.org/XML/1998/namespace
xs     http://www.w3.org/2001/XMLSchema               Namespace only, no code
>>

17 Code Generation Tools
========================

Main work horse of code generation is xsd2sg.pl, which serves multiple
purposes

1. Build hashes of all declarations in .sg input. Each hash element consists
   of array of elements and attributes, as well as groups and attribute groups.
   The type of array element sis determined from prefix, per .sg rules.
2. Expand groups and attribute groups
3. Evaluate each element wrt its type and generate
   a. C data structures
   b. Decoder grammar
   c. Token descriptions for perfect hash and lexical analyzer
   d. Encoder C code

The code to build hashes is interwoven in the code that generates .xsd
from .sg. The rest of the generation happens in a function called
generate().

Typical command line (to generate SAML 2.0 protocol engine)

  ~/plaindoc/xsd2sg.pl -d -gen saml2 -p zx_ \
       -r saml:Assertion -r se:Envelope \
       -S \
       sg/saml-schema-assertion-2.0.sg \
       sg/saml-schema-protocol-2.0.sg \
       sg/xmldsig-core.sg \
       sg/xenc-schema.sg \
       sg/soap11.sg \
       >/dev/null

<<ignore: ~/plaindoc/xsd2sg.pl -d -gen saml2 -p zx_ -r saml:Assertion -r se:Envelope -S sg/saml-schema-assertion-2.0.sg sg/saml-schema-protocol-2.0.sg sg/xmldsig-core.sg sg/xenc-schema.sg sg/soap11.sg >/dev/null >>

To generate SAML 2.0 Metadata engine you would issue

  ~/plaindoc/xsd2sg.pl -d -gen saml2md -p zx_ \
       -r md:EntityDescriptor -r md:EntitiesDescriptor \
       -S \
       sg/saml-schema-assertion-2.0.sg \
       sg/saml-schema-metadata-2.0.sg \
       sg/xmldsig-core.sg \
       sg/xenc-schema.sg \
       >/dev/null

<<ignore: ~/plaindoc/xsd2sg.pl -d -gen saml2md -p zx_ -r md:EntityDescriptor -r md:EntitiesDescriptor -S sg/saml-schema-assertion-2.0.sg sg/saml-schema-metadata-2.0.sg sg/xmldsig-core.sg sg/xenc-schema.sg >/dev/null >>

17.1 Special Support for Specific Programming Languages
-------------------------------------------------------

While C code generation is the main output, and this can always be
converted to other languages using SWIG, sometimes a more natural
language interface can be built by directly generating it.

We plan to enhance the code generation to do something like this. At
least direct hash-of-hashes-of-arrays-of-hashes type data-structure
generation for benefit of some scripting languages is planned.

18 ZXID SP
==========

*** warning: not checked lately, may be wrong!

<<table: ZXID SP URLs
URL          Description
============ =======================================================
/zxid        Same as o=M. Main convenience entry point
/zxid?o=M    SSO with CDC; or management if already logged in
/zxid?o=C    Common Domain Cookie (CDC) reader, usually under common domain host name.
/zxid?o=E    SSO after CDC read; or management if already logged in.
/zxid?o=P    HTTP POST end point. Used for forms and last part of POST profile SSO.
/zxid?o=S    SOAP end point (HTTP POST)
/zxid?o=B    Get SP metadata (or combined SP and IdP metadata if proxying).
>>

*** add description of CGI fields

19 Certificates
===============

*** TBD - This chapter should be elaborated to be a certificate tutorial with
following contents:

* Intro to certs and private keys
* Generating self signed cert
* Generating certificate signing request and using it to obtain
  commercially issued cert
* Installing root certs so you can recognize other people's certs
* Client TLS considerations

For the time being, the short answer is that ZXID uses OpenSSL and
PEM format certificates. You can use same techniques as you would use for
Apache / mod_ssl for acquiring certificates.

You should NEVER password protect your private key. There will not
be any opportunity to supply the password. You should instead protect
your private key using Unix filesystem permissions. See OpenSSL.org
or modssl.org FAQs for further information, including how to remove
a password if you accidentally enabled it.

96 License
==========

Copyright (c) 2006-2007 Symlabs (symlabs@symlabs.com), All Rights Reserved.
Author: Sampo Kellomäki (sampo@iki.fi)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

While the source distribution of ZXID does not contain
SSLeay or OpenSSL code, if you use this code you will use OpenSSL
library. Please give Eric Young and OpenSSL team credit (as required by
their licenses).

Binary distribution of this product includes software developed by the
OpenSSL Project for use in the OpenSSL Toolkit
(http://www.openssl.org/). See LICENSE.openssl for further information.

Binary distribution of this product includes cryptographic software
written by Eric Young (eay@cryptsoft.com).  Binary distribution of
this product includes software written by Tim Hudson
(tjh@cryptsoft.com). See LICENSE.ssleay for further information.

And remember, you, and nobody else but you, are responsible for
auditing ZXID and OpenSSL library for security problems,
back-doors, and general suitability for your application.

96.1 Dependency library licenses
--------------------------------

ZXID strives to maintain IPR hygiene and avoid both
non-free and GPL license contamination. All the
dependency libraries have BSD style licenses

* OpenSSL under BSDish (with "advertising" clause)
* libcurl under BSDish
* zlib under BSDish
* libc available as part of the operating system

Please see each library package for the exact details of their
licenses.

96.2 Specification IPR
----------------------

ZXID is based on open SAML and Liberty specifications. The parties
that have developed these specifications, including Symlabs, have made
Royalty Free (RF) licensing commitment. Please ask OASIS and Liberty
Alliance for the specifics of their IPR policies and IPR disclosures.

Some protocols, such as WS-Trust and WS-Federation enjoy Microsoft's
pledge<<footnote: If you have a reference to where this pledge can be
found, please let me know so it can be included here.>> that they will
not sue you even if you implement these specifications. You should
evaluate yourself whether this is good enough for your situation.

96.3 Further Warranties
-----------------------

If you need the author or Symlabs to further disclaim IPR interest or
make warranties of non-infringement, such declarations are
available for a fee. Please contact sales@symlabs.com

Legal queries and clarifications will be answered at then-current
Symlabs Professional Services rate, please contact sales@symlabs.com.

97 FAQ
======

*** real user FAQs are still lacking. Maybe this stuff is perfect?

97.1 Compilation Problems
-------------------------

97.1.1 Missing gperf
~~~~~~~~~~~~~~~~~~~~

  gcc -g -fpic -fmessage-length=0 -Wno-unused-label -Wno-unknown-pragmas -fno-strict-aliasing -D_REENTRANT -DDEBUG -DUSE_CURL -DUSE_OPENSSL -DLINUX -I/c/cvs/zxid_cvs -I/usr/local/ssl/include -I/usr//include   -c -o c/zx-a-aux.o c/zx-a-aux.c
  c/zx-a-aux.c: In function "zx_NEW_a_Action":
  c/zx-a-aux.c:80: error: "zx_a_Action_ELEM" undeclared (first use in this function)

This happens because c/zx-const.h was misgenerated (it should not happen at
all if you do not supply ENA_GEN=1) and does not include the necessary defines.
c/zx-const.h should have more then 1900 lines and look something like

  /* generated file, do not edit! zx_ _ATTR */
  #ifndef _zx__ATTR
  #define _zx__ATTR
  #define zx_use_ATTR     0
  #define zx_used_ATTR    1
  #define zx_sequence_ATTR        2
  ...
  #define zx_wantDSEPR_ATTR       347
  #define zx_ZX_TOK_NOT_FOUND_ATTR        348
  #define zx__ATTR_MAX    349
  #endif
  /* generated file, do not edit! zx_ _ELEM */
  #ifndef _zx__ELEM
  #define _zx__ELEM
  #define zx_ds_Y_ELEM    0
  #define zx_gl_Y_ELEM    1
  #define zx_gl_esrd_ELEM 2
  ...
  #define zx_wst_OnBehalfOf_ELEM  1629
  #define zx_ZX_TOK_NOT_FOUND_ELEM        1630
  #define zx__ELEM_MAX    1631
  #endif

97.1.2 Perl compiled with different compiler than zxid
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Perl modules generally want to be compiled with the same C compiler
and options as were used to compile perl itself (see perl -V). If this
happens to be different than the compiler you have defined in CC
variable (gcc by default, near top of Makefile or in localconf.mk), you may
get an error like:

  cd Net; perl Makefile.PL && make
  Warning: -L.. changed to -L/home/sampo/zxid/Net/..
  Writing Makefile for Net::SAML
  make[1]: Entering directory `/home/sampo/zxid/Net'
  cc -c  -I.. -I/apps/openssl/std/include -I/apps/include -D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -g -Wall -pipe   -DVERSION=\"\" -DXS_VERSION=\"\" -fPIC "-I/usr/lib/perl5/5.8.8/i586-linux-thread-multi/CORE"   SAML_wrap.c
  /bin/sh: cc: command not found
  make[1]: *** [SAML_wrap.o] Error 127
  make[1]: Leaving directory `/zxid/Net'
  make: *** [samlmod] Error 2

*Solutions*

1. Compile zxid with compiler that was used for perl, e.g.

     make CC=the-compiler-that-perl-wants

2. Recompile perl using the compiler that you want to use for zxid
3. Tinker with PATH environment variable so that both C compilers
   are found. However, using two different compilers is not really supported.

In general these types of problems happen when you use perl installed
by your distribution, but have later compiled a gcc of your own. It may
even be that you never installed the distribution cc - in that case
consider installing it and then trying approaches 1 or 3.

A similar situation can arise with incompatibility of the compiler and
options used for dependency libraries, such as OpenSSL or libcurl, and
those used for compiling zxid itself.

97.2 Common Mistakes
--------------------

1. Single Logout does not end the IdP session (i.e. IdP does
   not force you to supply password when you do SSO next time).

   Usual cause is that the management form (the one with the SLO buttons)
   does not have correct or any session ID. Do a view source on the
   the page and look for field called "s". The session ID is
   supposed to be extracted from the Single Sign-On result. For
   zxid_simple() you need to parse the returned LDIF and
   take the sesid. Pass that to zxid_fed_mgmt() as second argument.

2. Login buttons do nothing.

   A possible cause is that the entity ID is not passed from
   the IdP selection form. If the form is using POST method,
   you must make sure you actually read the HTTP body and
   pass its contents to the zxid_simple() as the qs argument.

3. The SP Login, a.k.a. IdP selection, page shows, but SSO does not work

   a. Your configuration does not match actual URL used to
      access the zxid system. For the zxidhlo family of
      examples you MUST edit the configuration string
      to match your situation. Watch out for domain name
      and port number.

   b. Connectivity issue prevents IdP from fetching metadata.
      Make sure your domain name is resolvable at IdP (e.g.
      add it to /etc/hosts). See also next point.

   c. IdP is not configured to get your metadata automatically.
      You have to configure your metadata to the IdP manually.
      How to do this depends on IdP product. Do not ask us.

   d. You supplied IdP URL that, in fact, is not the well known
      location for fetching IdP metadata. Or the IdP does
      not have well known location enabled. In the latter
      case you will need to install the IdP metadata
      manually (*** procedure to be documented). See [SAML2Meta]
      section 4.1 "Publication and Resolution via Well-Known Location",
      p.29, for normative description of this method.

   e. Connectivity issue at web browser level. Make sure your
      web browser can resolve both SP and IdP domain names.
      Edit /etc/hosts as needed.

   f. Personal firewall blocks access. Check firewall set up on
      * browser machine
      * SP machine
      * IdP machine

4. The SP Login, a.k.a. IdP selection, page does not show at all

   a. Connectivity issue at web browser level. Make sure your
      web browser can resolve both SP and IdP domain names.
      Edit /etc/hosts as needed.

   b. Personal firewall blocks access. Check firewall set up on
      * browser machine
      * SP machine

   c. You deployed the zxid in some other URL than you thought.
      Double check your webserver or Servlet container
      configuration and be sure you understand where
      zxid is supposed to appear. Be sure you are editing
      the right configuration - some people run multiple
      web servers in their machine and get confused about
      which one actually is active on which port and where
      the configuration files are located.

   d. ZXID lacks execute permissions or CGI permission
      setup prevents it from running. See previous bullet.

4. Mystery configuration problems. Double check /var/zxid/zxid.conf
   or consider removing it if you do not understand what it does.
   Double check the conf string if using zxid_simple() interface.

97.3 Consent
------------

A frequent concern among the business people and lawyer types is
whether the architecture provides for consent by the user. Usually
this is related to (avoidance of) liability and if the system can be
said to have gathered the consent of the user, we are safe.

Unfortunately the standards do not mandate an uniform user interface,
thus there is no single specific way how the consent is gathered or
determined: it depends from business situation and application to
another. Fortunately the Liberty and SAML 2.0 architectures provide
plenty of ways and hooks to gather and convey the consent. Consider the
following:

1. When arriving to SP, user chooses IdP for SSO. This act
   of course manifests user's intent to perform SSO.

2. IdP can ask the user whether he wants to perform SSO to
   the SP (IdP can make this question even if user is already
   logged in to the IdP, though most demos omit the question
   in the already logged in case).

   At this point the IdP may also ask whether the user
   wants to create a federation so that the SP can track
   the user. Creating federation is consenting to be
   tracked by the SP.

   If the federation already exists, the IdP can still
   offer a choice: should the federation be used
   this time, i.e. does the user consent to be tracked
   this time specifically.

   If user does not consent to federation and use of
   federation this time, but still consents to SSO,
   the SSO will be made using a temporary name ID.

3. If user gives any Personally Identifying Information
   to the SP (beyond the federated pseudonym), then
   the SP may be able to "connect the dots" and
   correlate user's actions on the SP with his actions
   in some other systems (technically this is called
   collusion).

   In a very technical sense users should be aware
   of this risk or the implication and therefore by
   providing such information they are effectively
   consenting to be correlated across systems.

   However, lawyers would probably say that if
   the SP intends to correlate, it should state
   so to the user at the time the information is
   asked so that the user can make an informed decision.
   If, after being informed, the user still supplies
   the information, then user is clearly
   consenting to the information being used for
   the stated purpose, i.e. correlation.

4. When user starts to use an ID web service, the user is
   consenting to this service being visible to at least
   some parties (why use the service if you did not
   intend this).

   To make this consent explicit, the user interface
   of the ID Web Service can ask.

   Also, the Discovery Service can ask consent using
   the Liberty Interaction Service. It is quite
   appropriate for the DS to ask this consent
   because it allows the ACL to be set correctly
   right from the beginning, when the service is
   registered.

5. When the user later accesses an SP that needs to
   contact an ID Web Service, it could be
   construed that the user, by using the SP
   at all, is effectively consenting that the SP
   may access the ID Web Services of the user.

   If this is not enough, the Discovery Service
   can use the Interaction Service on per service
   invocation basis to ask if the user consents
   to the specific request.

   Finally, the actual ID Web Service can also
   invoke the Interaction Service to ask the
   user to consent to the specific request,
   or otherwise enforce its policies.

6. When using People Service, the inviter (Alice)
   consents to the access by the invitee (Bob)
   by requesting an invitation string from the
   system.

   Once the invitation has been sent (and
   accepted by invitee) there is no easy way
   to collect consent from inviter on per
   request basis. For example Alice may not
   be online at the time when Bob accesses
   her resource.

   Alice can later revoke Bob's invitation,
   but in the window between Alice sending
   the invite and revoking it, Bob can access
   Alice's resource without Alice actively
   consenting to every access.

   Of course the resource can implement ACL
   policies, like only allowing Bob to access
   the resource a limited number of times,
   such as once.

7. When the invitee (Bob) uses inviter's (Alice's)
   ID Web Services (resources), Bob has
   consented to some form of tracking by
   Alice's resources by accepting the
   invite. Further consent may be obtained
   by Bob's own IdP, see bullet 2.

97.4 Use of Signing and Crypto, Security Concerns
-------------------------------------------------

97.5 Vendor products
--------------------

97.5.1 Symlabs Federated Identity Access Manager (FIAM)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Metadata import to IdP?

What I usually do is

  cd /opt/SYMfiam/3.0.x/conf/symdemo-idpa
  echo 'sp: zxid-sp1$https://sp1.zxidsp.org:8443/zxid?o=B$$' >>cot.ldif

Double check with text editor that the file is sensible.
Note that the single quotes are essential as the dollars
are to be interpretted literally, as separators.

  cd pem
  wget https://sp1.zxidsp.org:8443/zxid?o=B >zxid-sp1.xml

Here the intent is to fetch the metadata from the SP and
store it in a file whose name (without .xml extension)
matches the first component of the sp: line. I am not
100% on the wget syntax. You can also use browser
to fetch the metadata and simply Save as under the
correct name.

  cd /opt/SYMfiam/3.0.x/conf/symdemo-idpa/start.sh restart

This should restart the IdP server process and cause a
refresh of the metadata it may have cached. You may
want to

  tail -f /opt/SYMfiam/3.0.x/conf/symdemo-idpa/log/debug.log

to see if its getting indigestion.

N.B. FIAM seems to have NameID encryption on by default, Turn
this off by editing slimidp.ldif:

  encnids: 0

If this is not done, the SSO will fail (with what appears like signature
error).1x

97.6 Known Bugs
---------------

Following are known limitations. We document them here
because we do not plan to fix them in foreseeable future.

1. Unknown XML attributes are not sorted according to rules
   of exc-c14n. Instead they appear always +after+ known
   XML attributes and in the order they happen to be
   in the linked list.

   *Work around:* Add the attribute to schema (.sg) and
   regenerate and rebuild.

2. Encrypted NameIDs (or EncryptedAssertion) are not supported

3. Namespace qualified XML attributes have underscore instead of colon

97.7 Mysterious Error Messages
------------------------------

"Random number generator not seeded!!!"

This warning indicates that randomize() was not able to read
/dev/random or /dev/urandom, possibly because your system does not
have them or they are differently named. You can still use SSL, but
the encryption will not be as strong. Investigate setting up
EGD (entropy gathering daemon) or PRNG (Pseudo Random Number
Generator). Both are available on the net.

"msg 123: 1 - error:140770F8:SSL routines:SSL23_GET_SERVER_HELLO:unknown proto"

SSLeay error string. First number (123) is PID, second number (1) indicates
the position of the error message in SSLeay error stack. You often see
a pile of these messages as errors cascade.

"msg 123: 1 - error:02001002::lib(2) :func(1) :reason(2)"

The same as above, but you didn't call load_error_strings() so SSLeay
couldn't verbosely explain the error. You can still find out what it
means with this command:

     /usr/local/ssl/bin/ssleay errstr 02001002

97.7.1 Password is being asked for private key
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is normal behaviour if your private key is encrypted. Either
you have to supply the password or you have to use unencrypted
private key. Scan OpenSSL.org for the FAQ that explains how to
do this.

97.7.2 snprintf() multibyte character related errors in log
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is due to locale setting. Try

  export LANG=C

This will disable any UTF-8 processing in sprintf().

97.8 Author's Pet Peeves
------------------------

1. What is Schema Grammar (.sg) and why are you using it?
   * Schema Grammar is a compact formal description of XML documents. It is
     mostly bidirectionally convertible to XML Schema (XSD) and captures
     the useful essence of most XML schemata.
   * Schema Grammars are intuitive and compact, often allowing the
     essence to be understood at glance, and even most complex cases
     being only about 50% of the volume of the corresponding XSD.
   * We use Schema Grammar descriptions because they are more human readable
     than XSD and still equally amenable to automated code generation.
   * Schema Grammar descriptions are usually converted using xsd2sg.pl, which is
     part of the PlainDoc distribution.
   * See http://mercnet.pt/plaindoc
   * N.B. You do not need xsd2sg.pl or PlainDoc if you just want to compile and use ZXID.

2. What is PlainDoc (.pd)?
   * PlainDoc is a document preparation system that uses intuitive plain text files
     with minimal markup to generate PDF and HTML outputs.
   * We use PlainDoc because it makes it easy to maintain documentation.
   * See http://mercnet.pt/plaindoc
   * N.B. You do not need PlainDoc if you just want to compile and use ZXID.

3. How come zxid is so heavy to compile?
   * SAML 2.0 and related specs have a lot of functionality and detail, even
     if you really only need 1% of it. We do not wish to arbitrate which
     functionality is best or most needed, so we simply provide it all.
   * A lot of the code is generated, thus the input for C compiler is well
     in excess of half a million lines of code (of which only about 6k
     were written by a human).
   * Some of the generated files are gigantic, e.g. Net/SAML/zxid_wrap.c
     is over 380k lines. Compiler has to process all of this as a single
     compilation unit.
   * gcc and gnu ld were, perhaps, not designed to process this large inputs
     efficiently. Often the implementation strategy of keeping
     everything in memory will cause a smaller machines to swap.
   * My 1GHz CPU, 256 MB RAM machine definitely swaps and thus
     takes about 45 minutes to compile all this stuff.
   * I recommend at least 1GB RAM and 3GHz CPU for development
     machine. On such machine, you should be able to build in about 10 min.

4. Why do you not use ./configure and GNU autoconf?
   * ~autoconf~ is not for everyone. World does not stop without
     ~autoconf~. Or indeed need ~autoconf~. It is Yet Another Dependency
     I Do Not Need (YADIDNN).
   * I find the GNU ~autoconf~ stuff much more difficult to understand than
     my own ~Makefile~. Why should I debug ~autoconf~ when I could
     spend the time debugging my ~Makefile~ or the actual code?
   * I find resolving problems much easier at source code and ~Makefile~ level
     than trying to debug a million line script generated by some system
     I do not understand (perhaps some hardcore ~autoconf~ advocate could
     try to convince me and educate me, but I doubt).
   * My policy is to only support systems I have first hand experience with,
     or I have trustworthy friends to rely on. It does not help me
     to have a system that tries to guess +gazillion irrelevant variables+
     to an unpredictable state. It's much easier to stick to standards like
     POSIX and make sure you have predictable results from predictable inputs.
   * If the deterministic and predictable results are wrong, they can
     at least be debugged and fixed with a finite amount of work.
   * Supporting all relevant systems manually is not that much of work. The
     inhabitants of the irrelevant systems can support themselves, probably
     learning a great deal on the side.

97.9 What does ZXID aim at - an answer
--------------------------------------

A recent conversation that touched on the aims of ZXID project:

> So just generally, what are your goals for it, are you interested in making
> it work well with what other people are producing (e.g. SAML -> WSF
> cross-over), etc? I'm certainly assuming the answer's yes to that.

I aim at full stack client side implementation. ID-FF, SAML 2.0,
WSF (both versions). The generation technique I use will yield
the encoders and decoders for both WSP and WSC, but the hand written
higher level logic will at first be only written for SP and WSC.
Some WSP support has now been written as well.

It is Apache licensed project, of course, so if someone contributes
the IdP and WSP capabilities, I'll merge them into the distribution.

I am interested to have it working with other people's code at 3 levels:

1. Over-the-wire iop
2. I have split the functionality of the SP from the WSC such that
   zxid SP could probably be used with someone else's WSC and someone
   else's SP would reasonably easily be able to use zxid WSC.
3. Interfaces to non IdM parts of the complete system, typically
   used to implement the application layer, shall be
   plentiful: C/C++ API, Net::SAML/mod_perl, php - whatever you
   can SWIGify.

One thing I am NOT interested in is "layered" stack. I strongly
believe it's better each vertically integrated slice is implemented by
one mind. Thus, except for lowest HTTP, TLS, and TCP/IP layers, my SP,
or WSC, or WSP, handles the whole depth of the stack - SOAP,
signature, and app interface layers (of course the actual app should
be its own layer and probably user written). That is by design.

I have found in practise that if you attempt a layered stack, you have
impedance mismatches between the modules at different layers because
they were designed and written by different minds. By having vertical
integration I avoid impedance mismatches. This is the reason why
monolithic TCP/IP implementations tend to be better than explicitly
layered, such as the streams approach.

Now, if someone else wanted to take my generated encoders and
decoders and use them as a "layer" in their layered stack, I guess
I would not have any issue. If you do that, please let me know
because I would have to commit to API stability at that layer.
I am willing to do that once there are real projects that depend
on it, but until then I still may redesign those APIs, after
all, I am at revision 0.4 :-)

In the end, it seems that ZXID is actually somewhat layered approach -
what I mean by "vertical integration" is that all the layers are
designed and controlled by the same mind.

> BTW, I gather that it's SAML 2.0 at the moment, which I can't offer any test
> capability for, but if you get to SAML 1.1, I'm happy to set up some kind of
> IdP test capability for that.

In SSO world SAML 1.1 and ID-FF 1.2 capabilities are definitely on the
road map. In ID-WSF world, I'll probably start with 2.0 DS-WSC (don't
we all) followed by ID-DAP WSC and then tackle 1.1 after
that.<<footnote: As of version 0.18, July 2007, both WSC and WSP roles
of ID-DAP as well as ID-HR-XML have been implemented. This means the
generic WSC and WSP support is there.>>

97.9 Annoyances and improvement ideas
-------------------------------------

There is a lot of commonality that is not leveraged, especially in the
way service end points are chose given the metadata.  The descriptors
are nearly identical so casting them to one should work.

Many of the SAML2 responses are nearly identical. Rather than
construct them fully formally, we could have just one "SAML any
response" function. Perhaps this could be supported by some schema
grammar level aliasing feature: if an element derives from base type
without adding anything at all of its own, we might as well only
generate code for the base type.

Namespace aliasing scheme would allow us to consider two versions of
schema the same. It seems to be fairly common that the schema
changes are so minor that there is no justification for two
different decoding engines.

98 Support
==========

98.1 Mailing list and forums
----------------------------

Mail the author until we get the list set up. Or volunteer a list :-)

98.2 Bugs
---------

Mail the author until we get bug tracking set up. Or volunteer.

98.3 Developer access
---------------------

We use CVS, but access needs to be manually configured and is not
anonymous. If you contribute significantly, I will bother. Others
can send patches (good way to show you are worthy of CVS access)
to me. I've heard some mixed experiences about open source
sites like sourceforge. If you run such site and want to
host ZXID Project, please contact me.

If you just always want the latest source: get the tar ball from
the downloads section. Trust me, this is still so much in flux
that only the tar ball snapshots are in any usable state. CVS
access just to get latest source would be pointless.

98.9 Commercial Support
-----------------------

Following companies provide consultancy and support contracts for
ZXID:

* symlabs.com

99 Appendix: Schema Grammars
============================

Large parts of ZXID code are generated from +schema grammars+ which
are a convenient notation for describing XML schmata. This chapter
gives a sampling of some schema grammars that are currently implemented and
distributed in the ZXID package. For fuller list, see sg subdirectory
of the distribution or schemata.pd file.

<<table: Schema grammar syntax
Construct     Description
============= ====================================================================
  ee          Bareword signifies an XML element
  @aa         At (@) prefix signifies an XML attribute
  %tt         Percent (%) prefix signifies a complexType
  &gg         Ampersand (&) prefix a signifies group
  &@ag        Ampersand and at (&@) prefix signifies attributeGroup
  xx -> %tt   Arrow (->) signifies reference to type that defines element or attribute
  xx: ... ;   Colon (:) means that the definition of type follows immediately
  ee          An element or attribute by itself means exactly one occurance is expected
  ee?         Question mark (?) means the element or attribute is optional
  ee*         Asterisk (*) means the element may appear from zero to infinite number of times (same as * in regular expressions)
  ee+         Plus (+) means the element must appear at least once, but may appear an infinite number of times (same as + in regular expressions)
  ee{x,y}     The element must appear between x and y times (same as in regex)
  ee | ee     The pipey symbol (|) means elements are mutually exclusive choices.
  ee ee       Concatenation of elements or attributes means sequence
  base( t )   Introduce Extension base type (derive a type)
  redef( .. ) Redefine a type (using <xs:redefine> construct)
  mixed(1)    Mark a complex type as having mixed content type, i.e. strings and elements alternate
  enum( ... ) Introduce enumeration of xs:strings
  any         xs:any, the XML arbitrary element extension mechanism
  @any        xs:anyAttribute, the XML arbitrary attribute extension mechanism
target( ... ) Define target namespace described by the schema
import( ... ) Bring in other schemata and namespaces
ns( ... )     Declare existence of another namespace (without importing it)
>>

<<tex: \small>>

99.1 SAML 2.0
-------------

99.1.1 saml-schema-assertion-2.0 (sa)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/saml-schema-assertion-2.0.sg>>
>>

99.1.2 saml-schema-protocol-2.0 (sp)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/saml-schema-protocol-2.0.sg>>
>>

99.1.4 saml-schema-metadata-2.0 (md)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/saml-schema-metadata-2.0.sg>>
>>

99.5 Liberty ID-WSF 2.0
-----------------------

99.5.1 liberty-idwsf-utility-v2.0 (lu)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-utility-v2.0.sg>>
>>

99.5.3 liberty-idwsf-soap-binding-v2.0 (b)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-soap-binding-v2.0.sg>>
>>

99.5.4 liberty-idwsf-security-mechanisms-v2.0 (sec)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-security-mechanisms-v2.0.sg>>
>>

99.5.5 liberty-idwsf-disco-svc-v2.0 (di)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-disco-svc-v2.0.sg>>
>>

99.5.7 id-dap (dap)
~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/id-dap.sg>>
>>

99.5.8 liberty-idwsf-subs-v1.0 (subs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-subs-v1.0.sg>>
>>

99.5.9 liberty-idwsf-dst-v2.1 (dst)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/liberty-idwsf-dst-v2.1.sg>>
>>

99.6 SOAP 1.1 Processor wsf-soap11 (e)
--------------------------------------

<<schema:
<<sg/wsf-soap11.sg>>
>>

99.7 XML and Web Services Infrastructure
----------------------------------------

99.7.1 xmldsig-core (ds)
~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/xmldsig-core.sg>>
>>

99.7.2 xenc-schema (xenc)
~~~~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/xenc-schema.sg>>
>>

99.7.3 ws-addr-1.0 (a)
~~~~~~~~~~~~~~~~~~~~~~

<<schema:
<<sg/ws-addr-1.0.sg>>
>>

100 Appendix: Some Example XML Blobs
====================================

These XML blobs are for reference. They have been pretty
printed. Indentation indicates nesting level and closing tags have
been abbreviated as "</>". The actual XML on wire generally does not
have any whitespace.

100.1 SAML 2.0 Artifact Response with SAML 2.0 SSO Assertion and Two Bootstraps
-------------------------------------------------------------------------------

This example corresponds to t/sso-w-bootstraps.xml in the distribution.

Both bootstraps illustrate SAML assertion as bearer token.

 <soap:Envelope
    xmlns:lib="urn:liberty:iff:2003-08"
    xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:wsa="http://www.w3.org/2005/08/addressing">
  <soap:Body>

    <sp:ArtifactResponse
        xmlns:sp="urn:oasis:names:tc:SAML:2.0:protocol"
        ID="REvgoIIlkzTmk-aIX6tKE"
        InResponseTo="RfAsltVf2"
        IssueInstant="2007-02-10T05:38:15Z"
        Version="2.0">
      <sa:Issuer
          xmlns:sa="urn:oasis:names:tc:SAML:2.0:assertion"
          Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
        https://a-idp.liberty-iop.org:8881/idp.xml</>
      <sp:Status>
        <sp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/></>

      <sp:Response
          xmlns:sp="urn:oasis:names:tc:SAML:2.0:protocol"
          ID="RCCzu13z77SiSXqsFp1u1"
          InResponseTo="NojFIIhxw"
          IssueInstant="2007-02-10T05:37:42Z"
          Version="2.0">
        <sa:Issuer
            xmlns:sa="urn:oasis:names:tc:SAML:2.0:assertion"
            Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
          https://a-idp.liberty-iop.org:8881/idp.xml</>
        <sp:Status>
          <sp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/></>

        <sa:Assertion
            xmlns:sa="urn:oasis:names:tc:SAML:2.0:assertion"
            ID="ASSE6bgfaV-sapQsAilXOvBu"
            IssueInstant="2007-02-10T05:37:42Z"
            Version="2.0">
          <sa:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
            https://a-idp.liberty-iop.org:8881/idp.xml</>

          <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
            <ds:SignedInfo>
              <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
              <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
              <ds:Reference URI="#ASSE6bgfaV-sapQsAilXOvBu">
                <ds:Transforms>
                  <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                  <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></>
                <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                <ds:DigestValue>r8OvtNmq5LkYwCNg6bsRZAdT4NE=</></></>
            <ds:SignatureValue>GtWVZzHYW54ioHk/C7zjDRThohrpwC4=</></>

          <sa:Subject>
            <sa:NameID
                Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent"
                NameQualifier="https://a-idp.liberty-iop.org:8881/idp.xml">PB5fLIA4lRU2bH4HkQsn9</>
            <sa:SubjectConfirmation
                Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
              <sa:SubjectConfirmationData
                  NotOnOrAfter="2007-02-10T06:37:41Z"
                  Recipient="https://sp1.zxidsp.org:8443/zxidhlo?o=B"/></></>

          <sa:Conditions
              NotBefore="2007-02-10T05:32:42Z"
              NotOnOrAfter="2007-02-10T06:37:42Z">
            <sa:AudienceRestriction>
              <sa:Audience>https://sp1.zxidsp.org:8443/zxidhlo?o=B</></></>

          <sa:Advice>

            <!-- This assertion is the credential for the ID-WSF 1.1 bootstrap (below). -->

            <sa:Assertion
                ID="CREDOTGAkvhNoP1aiTq4bXBg"
                IssueInstant="2007-02-10T05:37:42Z"
                Version="2.0">
              <sa:Issuer
                  Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
                https://a-idp.liberty-iop.org:8881/idp.xml</>
              <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                <ds:SignedInfo>
                  <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
                  <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
                  <ds:Reference URI="#CREDOTGAkvhNoP1aiTq4bXBg">
                    <ds:Transforms>
                      <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                      <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></>
                    <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                    <ds:DigestValue>dqq/28hw5eEv+ceFyiLImeJ1P8w=</></></>
                <ds:SignatureValue>UKlEgHKQwuoCE=</></>
              <sa:Subject>
                <sa:NameID/>  <!-- *** Bug here!!! -->
                <sa:SubjectConfirmation
                    Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"/></>
              <sa:Conditions
                  NotBefore="2007-02-10T05:32:42Z"
                  NotOnOrAfter="2007-02-10T06:37:42Z">
                <sa:AudienceRestriction>
                  <sa:Audience>https://sp1.zxidsp.org:8443/zxidhlo?o=B</></></></></>

          <sa:AuthnStatement
              AuthnInstant="2007-02-10T05:37:42Z"
              SessionIndex="1171085858-4">
            <sa:AuthnContext>
              <sa:AuthnContextClassRef>
                urn:oasis:names:tc:SAML:2.0:ac:classes:Password</></></>

          <sa:AttributeStatement>

            <!-- Regular attribute -->

            <sa:Attribute
                Name="cn"
                NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
              <sa:AttributeValue>Sue</></>

            <!-- ID-WSF 1.1 Bootstrap for discovery. See also the Advice, above. -->

            <sa:Attribute
                Name="DiscoveryResourceOffering"
                NameFormat="urn:liberty:disco:2003-08">
              <sa:AttributeValue>
                <di12:ResourceOffering
                    xmlns:di12="urn:liberty:disco:2003-08"
                    entryID="2">
                  <di12:ResourceID>
                    https://a-idp.liberty-iop.org/profiles/WSF1.1/RID-DISCO-sue</>
                  <di12:ServiceInstance>
                    <di12:ServiceType>urn:liberty:disco:2003-08</>
                    <di12:ProviderID>https://a-idp.liberty-iop.org:8881/idp.xml</>
                    <di12:Description>
                      <di12:SecurityMechID>urn:liberty:security:2005-02:TLS:Bearer</>
                      <di12:CredentialRef>CREDOTGAkvhNoP1aiTq4bXBg</>
                      <di12:Endpoint>https://a-idp.liberty-iop.org:8881/DISCO-S</></></>
                  <di12:Abstract>Symlabs Discovery Service Team G</></></></>

            <!-- ID-WSF 2.0 Bootstrap for Discovery. The credential (bearer token) is inline. -->

            <sa:Attribute
                Name="urn:liberty:disco:2006-08:DiscoveryEPR"
                NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
              <sa:AttributeValue>
                <wsa:EndpointReference
                    xmlns:wsa="http://www.w3.org/2005/08/addressing"
                    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
                    notOnOrAfter="2007-02-10T07:37:42Z"
                    wsu:Id="EPRIDcjP8ObO9In47SDjO9b37">
                  <wsa:Address>https://a-idp.liberty-iop.org:8881/DISCO-S</>
                  <wsa:Metadata xmlns:di="urn:liberty:disco:2006-08">
                    <di:Abstract>SYMfiam Discovery Service</>
                    <sbf:Framework xmlns:sbf="urn:liberty:sb" version="2.0"/>
                    <di:ProviderID>https://a-idp.liberty-iop.org:8881/idp.xml</>
                    <di:ServiceType>urn:liberty:disco:2006-08</>
                    <di:SecurityContext>
                      <di:SecurityMechID>urn:liberty:security:2005-02:TLS:Bearer</>

                      <sec:Token
                          xmlns:sec="urn:liberty:security:2006-08"
                          usage="urn:liberty:security:tokenusage:2006-08:SecurityToken">

                        <sa:Assertion
                            ID="CREDV6ZBMyicmyvDq9pLIoSR"
                            IssueInstant="2007-02-10T05:37:42Z"
                            Version="2.0">
                          <sa:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
                            https://a-idp.liberty-iop.org:8881/idp.xml</>
                          <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                            <ds:SignedInfo>
                              <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
                              <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
                              <ds:Reference URI="#CREDV6ZBMyicmyvDq9pLIoSR">
                                <ds:Transforms>
                                  <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                                  <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></>
                                <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                                <ds:DigestValue>o2SgbuKIBzl4e0dQoTwiyqXr/8Y=</></></>
                            <ds:SignatureValue>hHdUKaZ//cZ8UYJxvTReNU=</></>
                          <sa:Subject>
                            <sa:NameID
                                Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent"
                                NameQualifier="https://a-idp.liberty-iop.org:8881/idp.xml">
                              9my93VkP3tSxEOIb3ckvjLpn0pa6aV3yFXioWX-TzZI=</>
                            <sa:SubjectConfirmation
                                Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"/></>
                          <sa:Conditions
                              NotBefore="2007-02-10T05:32:42Z"
                              NotOnOrAfter="2007-02-10T06:37:42Z">
                            <sa:AudienceRestriction>
                              <sa:Audience>https://a-idp.liberty-iop.org:8881/idp.xml</></></>
                          <sa:AuthnStatement AuthnInstant="2007-02-10T05:37:42Z">
                            <sa:AuthnContext>
                              <sa:AuthnContextClassRef>
                                urn:oasis:names:tc:SAML:2.0:ac:classes:Password</></></></></></></></></></></></></></></></>

N.B. The AttributeStatement/Attribute/AttributeValue/
EndpointReference/Metadata/SecurityContext/
Token/Assertion/Conditions/AudienceRestriction/Audience is the same as
the IdP because in many products the IdP and Discovery Service roles
are implemented by the same entity. Note also that the audience of the inner
assertion is the discovery service where as the audience of the outer assertion
is the SP that will eventually call the Discovery Service.

100.2 ID-WSF 2.0 Call with X509v3 Sec Mech
------------------------------------------

 <e:Envelope
    xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:b="urn:liberty:sb:2005-11"
    xmlns:sec="urn:liberty:security:2005-11"
    xmlns:wsse="http://docs.oasis-open.org/wss/20 04/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
    xmlns:wsa="http://www.w3.org/2005/08/ addressing">
  <e:Header>
    <wsa:MessageID wsu:Id="MID">123</>
    <wsa:To wsu:Id="TO">...</>
    <wsa:Action wsu:Id="ACT">...</>
    <wsse:Security mustUnderstand="1">
      <wsu:Timestamp wsu:Id="TS"><wsu:Created>2005-06-17T04:49:17Z</></>
      <wsse:BinarySecurityToken
          ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3"
          wsu:Id="X509Token"
          EncodingType="http://docs.oas is-open.org/wss/2004/01/oasis-200401-wss-soap-message-securiy-1.0#Base64Binary">
        MIIB9zCCAWSgAwIBAgIQ...</>
      <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
        <ds:SignedInfo>
          <ds:Reference URI="#MID">...</>
          <ds:Reference URI="#TO">...</>
          <ds:Reference URI="#ACT">...</>
          <ds:Reference URI="#TS">...</>
          <ds:Reference URI="#X509">
            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
            <ds:DigestValue>Ru4cAfeBAB</></>
          <ds:Reference URI="#BDY">
            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
            <ds:DigestValue>YgGfS0pi56p</></></>
        <ds:KeyInfo><wsse:SecurityTokenReference><wsse:Reference URI="#X509"/></></>
        <ds:SignatureValue>HJJWbvqW9E84vJVQkjDElgscSXZ5Ekw==</></></></>
  <e:Body wsu:Id="BDY">
    <xx:Query/></></>

The salient features of the above XML blob are

* Signature that covers relevant SOAP headers and Body
* Absence of any explicit identity token.

Absence of identity token means that from the headers it is not
possible to identify the taget identity. The signature generally
coveys the Invoker identity (the WSC that is calling the
service). Since one WSC typically serves many principals, knowing
which principal is impossible.  For this reason X509 security mechanism is
seldom used in ID-WSF 2.0 world (with ID-WSF 1.1 the ResourceID
provides an alternative way of identifying the principal, thus making
X509 a viable option).

100.3 ID-WSF 2.0 Call with Bearer (Binary) Sec Mech
---------------------------------------------------

 <e:Envelope
    xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:b="urn:liberty:sb:2005-11"
    xmlns:sec="urn:liberty:security:2005-11"
    xmlns:wsse="http://docs.oasis-open.org/wss/20 04/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
    xmlns:wsa="http://www.w3.org/2005/03/ addressing">
  <e:Header>
    <wsa:MessageID wsu:Id="MID">...</>
    <wsa:To wsu:Id="TO">...</>
    <wsa:Action wsu:Id="ACT">...</>
    <wsse:Security mustUnderstand="1">
      <wsu:Timestamp wsu:Id="TS">
        <wsu:Created>2005-06-17T04:49:17Z</></>
      <wsse:BinarySecurityToken
          ValueType="anyNSPrefix:ServiceSess ionContext"
          EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64 Binary"
          wsu:Id="BST">
        mQEMAzRniWkAAAEH9RWir0eKDkyFAB7PoFazx3ftp0vWwbbzqXdgcX8fpEqSr1v4
        YqUc7OMiJcBtKBp3+jlD4HPUaurIqHA0vrdmMpM+sF2BnpND118f/mXCv3XbWhiL
        VT4r9ytfpXBluelOV93X8RUz4ecZcDm9e+IEG+pQjnvgrSgac1NrW5K/CJEOUUjh
        oGTrym0Ziutezhrw/gOeLVtkywsMgDr77gWZxRvw01w1ogtUdTceuRBIDANj+KVZ
        vLKlTCaGAUNIjkiDDgti=</>
      <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig #">
        <ds:SignedInfo>
          <ds:Reference URI="#MID">...</>
          <ds:Reference URI="#TO">...</>
          <ds:Reference URI="#ACT">...</>
          <ds:Reference URI="#TS">...</>
          <ds:Reference URI="#BST">...</>
          <ds:Reference URI="#BDY">
            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1 "/>
            <ds:DigestValue>YgGfS0pi56pu</></></>
        ...</></></>
  <e:Body wsu:Id="BDY">
    <xx:Query/></></>

100.4 ID-WSF 2.0 Call with Bearer (SAML) Sec Mech
-------------------------------------------------

 <e:Envelope
    xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:sb="urn:liberty:sb:2005-11"
    xmlns:sec="urn:liberty:security:2005-11"
    xmlns:wsse="http://docs.oasis-open.org/wss/20 04/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
    xmlns:wsa="http://www.w3.org/2005/08/addressing"
    xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
    xmlns:xenc="http://www.w3.org/2001/04/xmlenc#">
  <e:Header>
    <wsa:MessageID wsu:Id="MID">...</>
    <wsa:To wsu:Id="TO">...</>
    <wsa:Action wsu:Id="ACT">...</>
    <wsse:Security mustUnderstand="1">
      <wsu:Timestamp wsu:Id="TS">
        <wsu:Created>2005-06-17T04:49:17Z</></>

      <sa:Assertion
          xmlns:sa="urn:oasis:names:tc:SAML:2.0:assertion"
          Version="2.0"
          ID="A7N123"
          IssueInstant="2005-04-01T16:58:33.173Z">
        <sa:Issuer>http://idp.symdemo.com/idp.xml</>
        <ds:Signature>...</>
        <sa:Subject>
          <sa:EncryptedID>
            <xenc:EncryptedData>U2XTCNvRX7Bl1NK182nmY00TEk==</>
            <xenc:EncryptedKey>...</></>
          <sa:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"/></>
        <sa:Conditions
            NotBefore="2005-04-01T16:57:20Z"
            NotOnOrAfter="2005-04-01T21:42:4 3Z">
          <sa:AudienceRestrictionCondition>
            <sa:Audience>http://wsp.zxidsp.org</></></>
        <sa:AuthnStatement
            AuthnInstant="2005-04-01T16:57:30.000Z"
            SessionIndex="6345789">
          <sa:AuthnContext>
            <sa:AuthnContextClassRef>
              urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</></></>
        <sa:AttributeStatement>
          <sa:EncryptedAttribute>
            <xenc:EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element">
              mQEMAzRniWkAAAEH9RbzqXdgcX8fpEqSr1v4=</>
            <xenc:EncryptedKey>...</></></></>

      <wsse:SecurityTokenReference
          xmlns:wsse11="..."
          wsu:Id="STR1"
          wsse11:TokenType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0">
        <wsse:KeyIdentifier
            ValueType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLID">
          A7N123</></>

      <ds:Signature>
        <ds:SignedInfo>
          <ds:Reference URI="#MID">...</>
          <ds:Reference URI="#TO">...</>
          <ds:Reference URI="#ACT">...</>
          <ds:Reference URI="#TS">...</>
          <ds:Reference URI="#STR1">
            <ds:Transform Algorithm="...#STR-Transform">
              <wsse:TransformationParameters>
                <ds:CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/></></></>
          <ds:Reference URI="#BDY"/></>
        ...</></></>
  <e:Body wsu:Id="BDY">
    <xx:Query/></></>

*** is the reference above to wsse11:TokenType really correct?

Note how the <Subject> and the attributes are encrypted such that only
the WSP can open them. This protects against WSC gaining knowledge of
the NameID at the WSP.

<<references:

[SAMLCore11] SAML 1.1 Core, OASIS

[SAMLCore2] SAML 2.0 Core, OASIS 2005.

[SAML2Meta] Cantor, Moreh, Phipott, Maler, eds., "Metadata for the OASIS Security Assertion Markup Language (SAML) V2.0", OASIS 2005.

[xml-c14n] XML Canonicalization (non-exclusive), http://www.w3.org/TR/2001/REC-xml-c14n-20010315

[xml-exc-c14n] Exclusive XML Canonicalization, http://www.w3.org/TR/xml-exc-c14n/

[Disco2] Liberty ID-WSF Discovery service 2.0

[Disco12] Liberty ID-WSF Discovery service 1.1 (liberty-idwsf-disco-svc-v1.2.pdf)

[SecMech2] Liberty ID-WSF 2.0 Security Mechanisms

[SOAPAuthn2] Liberty ID-WSF 2.0 Authentication Service

[SOAPBinding2] Liberty ID-WSF 2.0 framework document that pulls together all aspects

[DST21] Liberty Data Services Template 2.1

[DST20] Liberty DST v2.0

[DST11] Liberty DST v1.1

[IDDAP] Liberty Identity based Directory Access Protocol

[IDPP] Liberty Personal Profile specification.

[Interact11] Liberty ID-WSF Interaction Service protocol 1.1

[FF12] Liberty ID Federation Framework 1.2, Protocols and Schemas

[SUBS2] Liberty Subscriptions and Notifications specification

[Schema1-2] Henry S. Thompson et al. (eds): XML Schema Part 1: Structures, 2nd Ed., WSC Recommendation, 28. Oct. 2004, http://www.w3.org/2002/XMLSchema

[XML] http://www.w3.org/TR/REC-xml

[RFC2251] LDAP

>>

<<htmlpreamble: <title>README ZXID</title><body bgcolor="#330033" text="#ffaaff" link="#ffddff" vlink="#aa44aa" alink="#ffffff"><font face=sans><h1>README ZXID</h1> >>

<<notapath: TCP/IP a.k.a xBSD/Unix n/a Perl/mod_perl PHP/mod_php Java/Tomcat>>
<<EOF: >>


Pointers from Pat
http://rnd.feide.no/2007/04/13/light-bulb-update-request-for-testing/
https://opensso.dev.java.net/public/extensions/index.html

wsu namespace missing from cached EPRs

Empty namespace prefix gives trouble

Sun XML does not parse

Add string and XML parsing based interface to create requests (and responses).

Add macros for OK response.
