diff --git a/.gitignore b/.gitignore index 1c431bc92..85b880e3a 100755 --- a/.gitignore +++ b/.gitignore @@ -226,6 +226,7 @@ build*/* \.gcno \.gcda +# gh pages # Contrib Folder contrib/Makefile diff --git a/gh-pages/docs/BP-Service-API.md b/gh-pages/docs/BP-Service-API.md index 95e2d45dc..73f83e2f4 100644 --- a/gh-pages/docs/BP-Service-API.md +++ b/gh-pages/docs/BP-Service-API.md @@ -756,6 +756,767 @@ In general there are two types of ZCO that is relevant to a user. ### File ZCO +* Can be used if you wish not to make a copy of the data in the ION SDR (possibly to save SDR space) +* You allow ION to access the original data (file) in the host computer's file system. Need to check if this restriction applies. +* See example in `bpdriver.c` +# BP Service API + +This tutorial goes over the basic user APIs for developing application software to take advantage of the Bundle Protocol (BP) services provided by ION. + +## Pre-requisite + +For a user-developed software to utilize BP services provided by ION, there are two pre-requisite conditions: (1) ION services must be properly configured and already running in a state ready to provide BP services, and (2) BP libraries and header files must be installed and linked to the user application. + +## Check ION Installation & BP Version + +A simple way to check if ION is installed in the host is to determine the installation location of ION, one can execute `which ionadmin` and it will show the directory where ionadmin is currently located: + +```bash +$ which ionadmin +/usr/local/bin/ionadmin +``` + +If ionadmin is not found, it means either ION was not built or not properly installed in the execution path. If ionadmin is found, run it and provide the command `v` at the `:` prompt to determine the installed ION software version. For example: + +```bash +$ ionadmin +: v +ION-OPEN-SOURCE-4.1.2 +``` + +Quit ionadmin by command `q`. Quitting `ionadmin` will not terminate any ION and BP services, it simply ends ION's user interface for configuration query and management. If you see warning messages such as those shown below when quitting `ionadmin`, then it confirms that ION is actually not running at the moment. There are no software error: + +```bash +: q +at line 427 of ici/library/platform_sm.c, Can't get shared memory segment: Invalid argument (0) +at line 312 of ici/library/memmgr.c, Can't open memory region. +at line 367 of ici/sdr/sdrxn.c, Can't open SDR working memory. +at line 513 of ici/sdr/sdrxn.c, Can't open SDR working memory. +at line 963 of ici/library/ion.c, Can't initialize the SDR system. +Stopping ionadmin. +``` + +You can also run `bpversion` to determine the version of Bundle Protocol built in the host: + +```bash +$ bpversion +bpv7 +``` + +## Determine BP Service State + +Once it is determined that ION has been installed, a user may want to determine whether BP service is running by checking for the presence of various BP daemons and shared memory/semaphores, which are created by ION for interprocess communications among the various the BP service daemons. + +To check if BP service is running, you can list current running processes using `ps -aux` command and inspect if you see the following BP service daemons: + +* `rfxclock` - background daemon that schedules ION network configuration events +* `bpclm` - background daemon that selects and meters data transmission to outbound convergence layers +* `bpclock` - background daemon that schedule BP level events +* `bptransit` - daemon to move inbound traffic to outbound queue for forwarding +* `ipnfw` - BP IPN scheme-specific bundle routing daemon +* `ipnadminep` - IPN scheme-specific administrative end-point daemon + +You can find more details about these daemons in the manual pages. You may also see daemons related to other activate modules. For example, if the LTP engine is active in the system, you will see the following daemons: + +* `ltpclock` +* `ltpdeliv` +* `ltpmeter` +* `udplso` +* `udplsi` +* `ltpcli` +* `ltpclo` + +To further vary that BP service is running, you can check for presence of ION shared memory and semaphores: + +```bash +------ Shared Memory Segments -------- +key shmid owner perms bytes nattch status +0x0000ee02 47 userIon 666 641024 13 +0x0000ff00 48 userIon 666 50000000 13 +0x93de0005 49 userIon 666 1200002544 13 +0x0000ff01 50 userIon 666 500000000 13 + +------ Semaphore Arrays -------- +key semid owner perms nsems +0x0000ee01 23 userIon 666 1 +0x18020001 24 userIon 666 250 +``` + +In this example, the shared memory and semaphore keys for the SDR heap space (shmid 49) and the semaphorebase (semid 24) are created using a random key generated from the process ID of `ionadmin` and they will vary each time ION is instantiated. This is specific to SVR4 semaphore, which is the default for ION 4.1.2. However, starting with ION 4.1.3, the default semaphore will switch to POSIX semaphore and the output will be different. The other memory and semaphore keys listed in this example are typical default values, but they too, can be changed through ionconfig files. + +### If ION is installed but not running + +If ION is installed but not running (either you see no shared memory or don't see the BP service daemons) you can restart ION. Before restarting ION, run `ionstop` to clear all remaining orphaned processes/shared memory allocations in case the previous instance of BP service was not properly shutdown or has suffered a crash. Here is an example of the output you may see: + +```bash +$ ionstop +IONSTOP will now stop ion and clean up the node for you... +cfdpadmin . +Stopping cfdpadmin. +bpv7 +bpadmin . +Stopping bpadmin. +ltpadmin . +Stopping ltpadmin. +ionadmin . +Stopping ionadmin. +bsspadmin . +BSSP not initialized yet. +Stopping bsspadmin. +This is a single-ION instance configuration. Run killm. +killm +Sending TERM to acsadmin lt-acsadmin acslist lt-acslist aoslsi lt-aoslsi aoslso lt-aoslso bibeadmin lt-bibeadmin bibeclo lt-bibeclo bpadmin lt-bpadmin bpcancel lt-bpcancel bpchat lt-bpchat bpclm lt-bpclm bpclock lt-bpclock bpcounter lt-bpcounter bpdriver lt-bpdriver bpecho lt-bpecho bping lt-bping bplist lt-bplist bpnmtest lt-bpnmtest bprecvfile lt-bprecvfile bpsecadmin lt-bpsecadmin bpsendfile lt-bpsendfile bpsink lt-bpsink bpsource lt-bpsource bpstats lt-bpstats bpstats2 lt-bpstats2 bptrace lt-bptrace bptransit lt-bptransit brsccla lt-brsccla brsscla lt-brsscla bsscounter lt-bsscounter bssdriver lt-bssdriver bsspadmin lt-bsspadmin bsspcli lt-bsspcli bsspclo lt-bsspclo bsspclock lt-bsspclock bssrecv lt-bssrecv bssStreamingApp lt-bssStreamingApp cgrfetch lt-cgrfetch cpsd lt-cpsd dccpcli lt-dccpcli dccpclo lt-dccpclo dccplsi lt-dccplsi dccplso lt-dccplso dgr2file lt-dgr2file dgrcli lt-dgrcli dgrclo lt-dgrclo dtka lt-dtka dtkaadmin lt-dtkaadmin dtn2admin lt-dtn2admin dtn2adminep lt-dtn2adminep dtn2fw lt-dtn2fw dtpcadmin lt-dtpcadmin dtpcclock lt-dtpcclock dtpcd lt-dtpcd dtpcreceive lt-dtpcreceive dtpcsend lt-dtpcsend file2dgr lt-file2dgr file2sdr lt-file2sdr file2sm lt-file2sm file2tcp lt-file2tcp file2udp lt-file2udp hmackeys lt-hmackeys imcadmin lt-imcadmin imcadminep lt-imcadminep imcfw lt-imcfw ionadmin lt-ionadmin ionexit lt-ionexit ionrestart lt-ionrestart ionsecadmin lt-ionsecadmin ionunlock lt-ionunlock ionwarn lt-ionwarn ipnadmin lt-ipnadmin ipnadminep lt-ipnadminep ipnd lt-ipnd ipnfw lt-ipnfw lgagent lt-lgagent lgsend lt-lgsend ltpadmin lt-ltpadmin ltpcli lt-ltpcli ltpclo lt-ltpclo ltpclock lt-ltpclock ltpcounter lt-ltpcounter ltpdeliv lt-ltpdeliv ltpdriver lt-ltpdriver ltpmeter lt-ltpmeter ltpsecadmin lt-ltpsecadmin nm_agent lt-nm_agent nm_mgr lt-nm_mgr owltsim lt-owltsim owlttb lt-owlttb psmshell lt-psmshell psmwatch lt-psmwatch ramsgate lt-ramsgate rfxclock lt-rfxclock sdatest lt-sdatest sdr2file lt-sdr2file sdrmend lt-sdrmend sdrwatch lt-sdrwatch sm2file lt-sm2file smlistsh lt-smlistsh smrbtsh lt-smrbtsh stcpcli lt-stcpcli stcpclo lt-stcpclo tcaadmin lt-tcaadmin tcaboot lt-tcaboot tcacompile lt-tcacompile tcapublish lt-tcapublish tcarecv lt-tcarecv tcc lt-tcc tccadmin lt-tccadmin tcp2file lt-tcp2file tcpbsi lt-tcpbsi tcpbso lt-tcpbso tcpcli lt-tcpcli tcpclo lt-tcpclo udp2file lt-udp2file udpbsi lt-udpbsi udpbso lt-udpbso udpcli lt-udpcli udpclo lt-udpclo udplsi lt-udplsi udplso lt-udplso amsbenchr lt-amsbenchr amsbenchs lt-amsbenchs amsd lt-amsd amshello lt-amshello amslog lt-amslog amslogprt lt-amslogprt amsshell lt-amsshell amsstop lt-amsstop bputa lt-bputa cfdpadmin lt-cfdpadmin cfdpclock lt-cfdpclock cfdptest lt-cfdptest bpcp lt-bpcp bpcpd lt-bpcpd ... +Sending KILL to the processes... +Checking if all processes ended... +Deleting shared memory to remove SDR... +Killm completed. +ION node ended. Log file: ion.log +``` + +At this point run `ps -aux` or `ipcs` to verify that ION has terminated completely. + +### Simple ION Installation Test + +When ION is not running, you can performance a simple unit test to verify ION is build properly. + +Navigate to the root directory of the ION source code, `cd` into the `tests` folder and then execute a `bping` test using the command: + +```bash +./runtest bping +``` + +You can watch the terminal output ION restarting itself and executing a loopback ping. When successful it will indicate at the end of the test: + +```bash +TEST PASSED! + +passed: 1 + bping + +failed: 0 + +skipped: 0 + +excluded by OS type: 0 + +excluded by BP version: 0 + +obsolete tests: 0 +``` + +## Locate ION Libraries and Header Files + +The standard `./configure; make; sudo make install; sudo ldconfig` process should automatically install the BP libraries under `/usr/local/lib` and the relevant header files under `/usr/local/include` unless ION is specifically configured to a different customized install location through the `./configure` script during the compilation process. Here is a list of libraries and header files you should find there: + +```bash +$ cd /usr/local/lib +$ ls +libamp.a libbp.so.0.0.0 libcgr.a libdtpc.la libmbedcrypto.so.7 libtc.so.0.0.0 libudpcla.a +libamp.la libbss.a libcgr.la libdtpc.so libmbedtls.a libtcaP.a libudpcla.la +libamp.so libbss.la libcgr.so libdtpc.so.0 libmbedtls.so libtcaP.la libudpcla.so +libamp.so.0 libbss.so libcgr.so.0 libdtpc.so.0.0.0 libmbedtls.so.14 libtcaP.so libudpcla.so.0 +libamp.so.0.0.0 libbss.so.0 libcgr.so.0.0.0 libici.a libmbedx509.a libtcaP.so.0 libudpcla.so.0.0.0 +libampAgentADM.a libbss.so.0.0.0 libdgr.a libici.la libmbedx509.so libtcaP.so.0.0.0 libudplsa.a +libampAgentADM.la libbssp.a libdgr.la libici.so libmbedx509.so.1 libtcc.a libudplsa.la +libampAgentADM.so libbssp.la libdgr.so libici.so.0 libstcpcla.a libtcc.la libudplsa.so +libampAgentADM.so.0 libbssp.so libdgr.so.0 libici.so.0.0.0 libstcpcla.la libtcc.so libudplsa.so.0 +libampAgentADM.so.0.0.0 libbssp.so.0 libdgr.so.0.0.0 libltp.a libstcpcla.so libtcc.so.0 libudplsa.so.0.0.0 +libams.a libbssp.so.0.0.0 libdtka.a libltp.la libstcpcla.so.0 libtcc.so.0.0.0 libzfec.a +libams.la libcfdp.a libdtka.la libltp.so libstcpcla.so.0.0.0 libtcpbsa.a libzfec.la +libbp.a libcfdp.la libdtka.so libltp.so.0 libtc.a libtcpbsa.la libzfec.so +libbp.la libcfdp.so libdtka.so.0 libltp.so.0.0.0 libtc.la libtcpbsa.so libzfec.so.0 +libbp.so libcfdp.so.0 libdtka.so.0.0.0 libmbedcrypto.a libtc.so libtcpbsa.so.0 libzfec.so.0.0.0 +libbp.so.0 libcfdp.so.0.0.0 libdtpc.a libmbedcrypto.so libtc.so.0 libtcpbsa.so.0.0.0 + + +$ cd /usr/local/include +$ ls +ams.h bpsec_instr.h cfdpops.h eureka.h llcv.h platform.h rfc9173_utils.h sdrhash.h sdrxn.h tcaP.h +bcb_aes_gcm_sc.h bpsec_util.h crypto.h icinm.h ltp.h platform_sm.h rfx.h sdrlist.h smlist.h tcc.h +bib_hmac_sha2_sc.h bss.h dgr.h ion.h lyst.h psa sci.h sdrmgt.h smrbt.h tccP.h +bp.h bssp.h dtka.h ion_test_sc.h mbedtls psm.h sda.h sdrstring.h sptrace.h zco.h +bpsec_asb.h cfdp.h dtpc.h ionsec.h memmgr.h radix.h sdr.h sdrtable.h tc.h +``` + +In this document, we assume that ION was build and installed via the `./configure` installation process using the full open-source codebase and with the standard set of options. + +The location and content of the library and header directories shown above included non-BP modules and may not match exactly with what you have especially if you have built ION with features options enabled via `./configure` or used a manual/custom Makefile, or built ION from the `ion-core` package instead. + +## Launch ION & BP Services + +Once you are confidant that ION has been properly built and installed in the system, you can start BP service by launching ION. To do this, please consult the various tutorials under _Configuration_. + +After launching ION, you can [verify BP service status](#determine-bp-service-state) in the same manner as described in previous section. + +## BP Service API Reference + +### Header + +```c +#include "bp.h" +``` + +--- + +### bp_attach + +Function Prototype + +```c +int bp_attach( ) +``` + +Parameters + +* None. + +Return Value + +* On success: 0 +* Any error: -1 + +Example Call + +```c +if (bp_attach() < 0) +{ + printf("Can't attach to BP.\n"); + /* user inser error handling code */ +} +``` + +Description + +Typically the `bp_attach()` call is made at the beginning of a user's application to attach to BP Service provided by ION in the host machine. This code checks for a negative return value. + +`bp_attach()` automatically calls the ICI API `ion_attach()` when necessary, so there is no need to call them separately. In addition to gaining access to ION's SDR, which is what `ion_attach()` provides, `bp_attach()` also gains access to the Bundle Protocol's state information and database. For user application that interacts with the Bundle Protocol, `bp_attach()` is the entry point to ION. + +--- + +### Sdr bp_get_sdr( ) + +Function Prototype + +```c +Sdr bp_get_sdr() +``` + +Parameters + +* None. + +Return Value + +* Handle for the SDR data store: success +* `NULL`: any error + +Example Call + +```c +/* declare SDR handle */ +Sdr sdr; + +/* get SDR handle */ +sdr = bp_get_sdr(); + +/* user check sdr for NULL + * and handle error */ +``` + +Description + +Returns handle for the SDR data store used for BP, to enable creation and interrogation of bundle payloads (application data units). Since the SDR handle is needed by many APIs, this function is typically executed early in the user's application in order to access other BP services. + +--- + +### bp_detach + +Function Prototype + +```c +void bp_detach( ) + +``` + +Parameters + +* None. + +Return Value + +* 0: success +* -1: Any error + +Description + +Terminates all access to BP functionality for the invoking process. + +--- + +### bp_open + +Function Prototype + +```c +int bp_open(char *eid, BpSAP *ionsapPtr) +``` + +Parameters + +* `*eid`: name of the endpoint +* `*ionsapPtr`: pointer to variable in which address of BP service access point will be returned + +Return Value + +* 0: success +* -1: any error + +Example Call + +```c +if (bp_open(ownEid, &sap) < 0) +{ + putErrmsg("bptrace can't open own endpoint.", ownEid); + + /* user's error handling function here */ +} +``` + +Description + +Opens the application's access to the BP endpoint identified by the string at `eid`, so that the application can take delivery of bundles destined for the indicated endpoint. This SAP can also be used for sending bundles whose source is the indicated endpoint. + +Please note that all bundles sent via this SAP will be subject to immediate destruction upon transmission, i.e., no bundle addresses will be returned by `bp_send` for use in tracking, suspending/resuming, or cancelling transmission of these bundles. + +On success, places a value in *ionsapPtr that can be supplied to future bp function invocations. + +__NOTE:__ To allow for bp_send to return a bundle address for tracking purpose, please use `bp_open_source` instead. + +--- + +### bp_open_source + +Function Prototype + +```c +int bp_open_source(char *eid, BpSAP *ionsapPtr, int detain) +``` + +Parameters + +* `*eid`: name of the endpoint +* `*ionsapPtr`: pointer to variable in which address of BP service access point will be returned +* `detain`: indicator as to whether or not bundles sourced using this BpSAP should be detained in storage until explicitly released + +Return Value + +* 0: success +* -1: Any error + +Example Call + +```c +if (bp_open_source(ownEid, &txSap, 1) < 0) +{ + putErrmsg("can't open own 'send' endpoint.", ownEid); + + /* user error handling routine here */ +} +``` + +Description + +Opens the application's access to the BP endpoint identified by eid, so that the application can send bundles whose source is the indicated endpoint. If and only if the value of detain is non-zero, citing this SAP in an invocation of bp_send() will cause the address of the newly issued bundle to be returned for use in tracking, suspending/resuming, or cancelling transmission of this bundle. + +__USE THIS FEATURE WITH GREAT CARE__: such a bundle will continue to occupy storage resources until it is explicitly released by an invocation of bp_release() or until its time to live expires, so bundle detention increases the risk of resource exhaustion. (If the value of detain is zero, all bundles sent via this SAP will be subject to immediate destruction upon transmission.) + +On success, places a value in *ionsapPtr that can be supplied to future bp function invocations and returns 0. Returns -1 on any error. + +--- + +### bp_send + +Function Prototype + +```c +int bp_send(BpSAP sap, char *destEid, char *reportToEid, + int lifespan, int classOfService, BpCustodySwitch custodySwitch, + unsigned char srrFlags, int ackRequested, + BpAncillaryData *ancillaryData, Object adu, Object *newBundle) +``` + +Parameters + +* `sap`: the source endpoint for the bundle, provided by the `bp_open` call. +* `*destEid`: identifies the destination endpoint for the bundle. +* `reportToEid`: identifies the endpoint to which any status reports pertaining to this bundle will be sent; if NULL, defaults to the source endpoint. +* `lifespan`: is the maximum number of seconds that the bundle can remain in-transit (undelivered) in the network prior to automatic deletion. +* `classOfService`: is simply priority for now: BP_BULK_PRIORITY, BP_STD_PRIORITY, or BP_EXPEDITED_PRIORITY. If class-of-service flags are defined in a future version of Bundle Protocol, those flags would be OR'd with priority. +* `custodySwitch`: indicates whether or not custody transfer is requested for this bundle and, if so, whether or not the source node itself is required to be the initial custodian. The valid values are SourceCustodyRequired, SourceCustodyOptional, NoCustodyRequired. Note that custody transfer is possible only for bundles that are uniquely identified, so it cannot be requested for bundles for which BP_MINIMUM_LATENCY is requested, since BP_MINIMUM_LATENCY may result in the production of multiple identical copies of the same bundle. Similarly, custody transfer should never be requested for a "loopback" bundle, i.e., one whose destination node is the same as the source node: the received bundle will be identical to the source bundle, both residing in the same node, so no custody acceptance signal can be applied to the source bundle and the source bundle will remain in storage until its TTL expires. +* `srrFlags`: if non-zero, is the logical OR of the status reporting behaviors requested for this bundle: BP_RECEIVED_RPT, BP_CUSTODY_RPT, BP_FORWARDED_RPT, BP_DELIVERED_RPT, BP_DELETED_RPT. +* `ackRequested`: is a Boolean parameter indicating whether or not the recipient application should be notified that the source application requests some sort of application-specific end-to-end acknowledgment upon receipt of the bundle. +* `ancillaryData`: if not NULL, is used to populate the Extended Class Of Service block for this bundle. The block's ordinal value is used to provide fine-grained ordering within "expedited" traffic: ordinal values from 0 (the default) to 254 (used to designate the most urgent traffic) are valid, with 255 reserved for custody signals. The value of the block's flags is the logical OR of the applicable extended class-of-service flags: + +``` +BP_MINIMUM_LATENCY designates the bundle as "critical" for the +purposes of Contact Graph Routing. + +BP_BEST_EFFORT signifies that non-reliable convergence-layer protocols, as +available, may be used to transmit the bundle. Notably, the bundle may be +sent as "green" data rather than "red" data when issued via LTP. + +BP_DATA_LABEL_PRESENT signifies whether or not the value of _dataLabel_ +in _ancillaryData_ must be encoded into the ECOS block when the bundle is +transmitted. +``` + +__NOTE:__ For Bundle Protocol v7, no Extended Class of Service, or equivalent, has been standardized yet. This capability, however, has been retained from BPv6 and is available to BPv7 implementation in ION. + +* `adu`: is the "application data unit" that will be conveyed as the payload of the new bundle. adu must be a "zero-copy object" (ZCO). To ensure orderly access to transmission buffer space for all applications, adu must be created by invoking ionCreateZco(), which may be configured either to block so long as insufficient ZCO storage space is available for creation of the requested ZCO or to fail immediately if insufficient ZCO storage space is available. + +Return Value + +* 1: success +* 0: user error +* -1: any system error + +Example Call + +```c +if (bp_send(sap, destEid, reportToEid, ttl, priority, + custodySwitch, srrFlags, 0, &ancillaryData, + traceZco, &newBundle) <= 0) +{ + putErrmsg("bptrace can't send file in bundle.", + fileName); + + /* user error handling code goes here */ +} +``` + +Description + +Sends a bundle to the endpoint identified by destEid, from the source endpoint as provided to the bp_open() call that returned sap. + +When sap is NULL, the transmitted bundle is anonymous, i.e., the source of the bundle is not identified. This is legal, but anonymous bundles cannot be uniquely identified; custody transfer and status reporting therefore cannot be requested for an anonymous bundle. + +The function returns 1 on success, 0 on user error, -1 on any system error. + +If 0 is returned, then an invalid argument value was passed to bp_send(); a message to this effect will have been written to the log file. + +If 1 is returned, then either the destination of the bundle was "dtn:none" (the bit bucket) or the ADU has been accepted and queued for transmission in a bundle. In the latter case, if and only if sap was a reference to a BpSAP returned by an invocation of bp_open_source() that had a non-zero value in the detain parameter, then newBundle must be non-NULL and the address of the newly created bundle within the ION database is placed in newBundle. This address can be used to track, suspend/resume, or cancel transmission of the bundle. + +--- + +### bp_track + +Function Prototype + +```c +int bp_track(Object bundle, Object trackingElt) +``` + +Parameters + +* `bundle`: the bundle object data structure +* `trackingElt`: an sdrlist element managed by the user's application + +Return Value + +* 0: success +* -1: any error + +Example Call + +```c +/* a lyst of bundles in SDR */ +Object bundleList; + +/* a bundle object in SDR */ +Object bundleObject; + +bundleElt = sdr_list_insert_last(sdr, bundleList, + bundleObject); +if (bp_track(outAdu.bundleObj, bundleElt) < 0) +{ + sdr_cancel_xn(sdr); + putErrmsg("Can't track bundle.", NULL); + + /* user error handling code goes here */ +} +``` + +The bundleList is managed via the [sdr_list](./ICI-API.md#sdr-list-management-apis) library of APIs. + +Description + +Adds `trackingElt` to the list of "tracking" references in bundle. `trackingElt` must be the address of an SDR list element -- whose data is the address of this same bundle -- within some list of bundles that is privately managed by the application. Upon destruction of the bundle this list element will automatically be deleted, thus removing the bundle from the application's privately managed list of bundles. This enables the application to keep track of bundles that it is operating on without risk of inadvertently de-referencing the address of a nonexistent bundle. + +--- + +### bp_untrack + +Function Prototype + +```c +void bp_untrack(Object bundle, Object trackingElt) +``` + +Parameters + +* `bundle`: the bundle object data structure +* `trackingElt`: an sdrlist element managed by the user's application + +Return Value + +* 0: success +* -1: Any error + +Description + +Removes `trackingElt` from the list of "tracking" references in bundle, if it is in that list. Does not delete `trackingElt` itself. + +--- + +### bp_suspend + +Function Prototype + +```c +int bp_suspend(Object bundle) +``` + +Parameters + +* `bundle`: a bundle object in the SDR + +Return Value + +* 0: success +* -1: Any error + +Description + +Suspends transmission of bundle. Has no effect if bundle is "critical" (i.e., has got extended class of service BP_MINIMUM_LATENCY flag set) or if the bundle is already suspended. Otherwise, reverses the enqueuing of the bundle to its selected transmission outduct and places it in the "limbo" queue until the suspension is lifted by calling bp_resume. Returns 0 on success, -1 on any error. + +--- + +### bp_resume + +Function Prototype + +```c +int bp_resume(Object bundle) +``` + +Parameters + +* `bundle`: a bundle object in the SDR + +Return Value + +* 0: success +* -1: Any error + +Description + +Terminates suspension of transmission of bundle. Has no effect if bundle is "critical" (i.e., has got extended class of service BP_MINIMUM_LATENCY flag set) or is not suspended. Otherwise, removes the bundle from the "limbo" queue and queues it for route re-computation and re-queuing. Returns 0 on success, -1 on any error. + +--- + +### bp_cancel + +Function Prototype + +```c +int bp_cancel(Object bundle) +``` + +Parameters + +* `bundle`: a bundle object in the SDR + +Return Value + +* 0: success +* -1: Any error + +Description + +Cancels transmission of bundle. If the indicated bundle is currently queued for forwarding, transmission, or retransmission, it is removed from the relevant queue and destroyed exactly as if its Time To Live had expired. Returns 0 on success, -1 on any error. + +--- + +### bp_release + +Function Prototype + +```c +int bp_release(Object bundle) +``` + +Parameters + +* `bundle`: a bundle object in the SDR + +Return Value + +* 0: success +* -1: Any error + +Description + +Releases a detained bundle for destruction when all retention constraints have been removed. After a detained bundle has been released, the application can no longer track, suspend/resume, or cancel its transmission. Returns 0 on success, -1 on any error. + +**NOTE**: for bundles sent through an bundle protocol end-point which is opened via `bp_open_source` with `detain` set to non-zero value, they will not be destroyed, even after successful transmissions, until time-to-live has expired or explicitly released via `bp_release`. + +--- + +### bp_receive + +Function Prototype + +```c +int bp_receive(BpSAP sap, BpDelivery *dlvBuffer, int timeoutSeconds) +``` + +Parameters + +* `sap`: the source endpoint for the bundle, provided by the `bp_open` call +* `*dlvBuffer`: a pointer to a `BpDelivery` structure used to return the received bundle and/or outcome of reception +* `timoutSeconds`: a reception timer in seconds + +Return Value + +* 0: success +* -1: any error + +Example Call + +```c +if (bp_receive(state.sap, &dlv, BP_BLOCKING) < 0) +{ + putErrmsg("bpsink bundle reception failed.", NULL); + + /* user code to handle error or timeout*/ +} +``` + +In this example, BP_BLOCKING is set to -1, that means that the call will block *forever* until a bundle is received, unless interrupted `bp_interrupt`. + +Description + +Receives a bundle, or reports on some failure of bundle reception activity. + +The "result" field of the dlvBuffer structure will be used to indicate the outcome of the data reception activity. + +If at least one bundle destined for the endpoint for which this SAP is opened has not yet been delivered to the SAP, then the payload of the oldest such bundle will be returned in `dlvBuffer->adu` and `dlvBuffer->result` will be set to BpPayloadPresent. If there is no such bundle, bp_receive() blocks for up to timeoutSeconds while waiting for one to arrive. + +If timeoutSeconds is BP_POLL (i.e., zero) and no bundle is awaiting delivery, or if timeoutSeconds is greater than zero but no bundle arrives before timeoutSeconds have elapsed, then `dlvBuffer->result` will be set to BpReceptionTimedOut. If timeoutSeconds is BP_BLOCKING (i.e., -1) then bp_receive() blocks until either a bundle arrives or the function is interrupted by an invocation of `bp_interrupt`(). + +`dlvBuffer->result` will be set to BpReceptionInterrupted in the event that the calling process received and handled some signal other than SIGALRM while waiting for a bundle. + +`dlvBuffer->result` will be set to BpEndpointStopped in the event that the operation of the indicated endpoint has been terminated. + +The application data unit delivered in the data delivery structure, if any, will be a "zero-copy object" reference. Use zco reception functions (see zco(3)) to read the content of the application data unit. + +Be sure to call `bp_release_delivery`() after every successful invocation of `bp_receive`(). + +The function returns 0 on success, -1 on any error. + +--- + +### bp_interrupt + +Function Prototype + +```c +void bp_interrupt(BpSAP sap) +``` + +Parameters + +* `bundle`: a bundle object in the SDR + +Return Value + +* 0: success +* -1: Any error + +Description + +Interrupts a `bp_receive`() invocation that is currently blocked. This function is designed to be called from a signal handler; for this purpose, sap may need to be obtained from a static variable. + +--- + +### bp_release_delivery + +Function Prototype + +```c +void bp_release_delivery(BpDelivery *dlvBuffer, int releaseAdu) +``` + +Parameters + +* `*dlvBuffer`: a pointer to a `BpDelivery` structure used to return the received bundle and/or outcome of reception +* `releaseAdu`: a Boolean parameter: if non-zero, the ADU ZCO reference in dlvBuffer (if any) is destroyed. + +Return Value + +* none + +Description + +Releases resources allocated to the indicated delivery by `dlvBuffer`, which is returned by bp_receive. `releaseAdu` is a Boolean parameter: if non-zero, the ADU ZCO reference in `dlvBuffer` (if any) is destroyed, causing the ZCO itself to be destroyed if no other references to it remain. + +--- + +### bp_close + +Function Prototype + +```c +void bp_close(BpSAP sap) +``` + +Parameters + +* `sap`: the source endpoint for the bundle, provided by the `bp_open` or `bp_open_source` call + +Return Value + +* none + +Description + +Terminates the application's access to the BP endpoint identified by the eid cited by the indicated service access point. The application relinquishes its ability to take delivery of bundles destined for the indicated endpoint and to send bundles whose source is the indicated endpoint. + +--- + +## Walk Through of `bpsource.c` + +* For this example, it is assumed that the user is already familiar with the [ICI APIs](./ICI-API.md). + +**TO BE UPDATED.** + +## Compiling and Linking + +## Zero-copy Object (ZCO) Types + +We have shown that the way to hand user data from an application to BP is via a zero-copy object (ZCO). + +In general there are two types of ZCO that is relevant to a user. + +* Note A third type of ZCO is called a ZCO ZCO - one consists of multiple other ZCOs. This is used internally by ION to handle concatenation of data but it is not relevant in the context of this topic. + +### SDR ZCO + +* Used in our example +* Example in `bpsource.c` + +### File ZCO + * Can be used if you wish not to make a copy of the data in the ION SDR (possibly to save SDR space) * You allow ION to access the original data (file) in the host computer's file system. Need to check if this restriction applies. * See example in `bpdriver.c` diff --git a/gh-pages/docs/ION-Guide.md b/gh-pages/docs/ION-Guide.md index a185aaa88..d67c8e94c 100644 --- a/gh-pages/docs/ION-Guide.md +++ b/gh-pages/docs/ION-Guide.md @@ -1795,6 +1795,3341 @@ RFC 5326.) **Figure 14 LTP link service input** +### Contact Graph Routing (CGR) + +CGR is a dynamic routing system that computes routes through a +time-varying topology of scheduled communication contacts in a DTN +network. It is designed to support operations in a space network based +on DTN, but it also could be used in terrestrial applications where +operation according to a predefined schedule is preferable to +opportunistic communication, as in a low-power sensor network. + +The basic strategy of CGR is to take advantage of the fact that, since +communication operations are planned in detail, the communication routes +between any pair of "bundle agents" in a population of nodes that have +all been informed of one another's plans can be inferred from those +plans rather than discovered via dialogue (which is impractical over +long-one-way-light-time space links). + +#### **Contact Plan Messages** + +CGR relies on accurate contact plan information provided in the form of +contact plan messages that currently are only read from **ionrc** files +and processed by **ionadmin**, which retains them in a non-volatile +contact plan in the RFX database, in ION's SDR data store. + +Contact plan messages are of two types: *contact messages* and *range +messages*. + +Each contact message has the following content: + +- The starting UTC time of the interval to which the message pertains. +- The stop time of this interval, again in UTC. +- The Transmitting node number. +- The Receiving node number. +- The planned rate of transmission from node A to node B over this + interval, in bytes per second. + +Each range message has the following content: + +- The starting UTC time of the interval to which the message pertains. +- The stop time of this interval, again in UTC. +- Node number A. +- Node number B. +- The anticipated distance between A and B over this interval, in + light seconds. + +Note that range messages may be used to declare that the "distance" in +light seconds between nodes A and B is **different** in the B🡪A +direction from the distance in the A🡪B direction. While direct radio +communication between A and B will not be subject to such asymmetry, +it's possible for connectivity established using other convergence-layer +technologies to take different physical paths in different directions, +with different signal propagation delays. + +#### **Routing Tables** + +Each node uses Range and Contact messages in the contact plan to build a +\"routing table\" data structure. + +The routing table constructed locally by each node in the network is a +list of *entry node lists*, one route list for every other node D in the +network that is cited in any Contact or Range in the contact plan. Entry +node lists are computed as they are needed, and the maximum number of +entry node lists resident at a given time is the number of nodes that +are cited in any Contacts or Ranges in the contact plan. Each entry in +the entry node list for node D is a list of the neighbors of local node +X; included with each entry of the entry node list is a list one or more +routes to D through the indicated neighbor, termed a *route lis*t. + +Each route in the route list for node D identifies a path to destination +node D, from the local node, that begins with transmission to one of the +local node's neighbors in the network-- the initial receiving node for +the route, termed the route's *entry node*. + +For any given route, the contact from the local node to the entry node +constitutes the initial transmission segment of the end-to-end path to +the destination node. Additionally noted in each route object are all of +the other contacts that constitute the remaining segments of the route's +end-to-end path. + +Each route object also notes the forwarding *cost* for a bundle that is +forwarded along this route. In this version of ION, CGR is configured to +deliver bundles as early as possible, so best-case final delivery time +is used as the cost of a route. Other metrics might be substituted for +final delivery time in other CGR implementations. NOTE, however, that if +different metrics are used at different nodes along a bundle's +end-to-end path it becomes impossible to prevent routing loops that can +result in non-delivery of the data. + +Finally, each route object also notes the route's termination time, the +time after which the route will become moot due to the termination of +the earliest-ending contact in the route. + +### **Key Concepts** + +#### **Expiration time** + +Every bundle transmitted via DTN has a time-to-live (TTL), the length of +time after which the bundle is subject to destruction if it has not yet +been delivered to its destination. The *expiration time* of a bundle is +computed as its creation time plus its TTL. When computing the next-hop +destination for a bundle that the local bundle agent is required to +forward, there is no point in selecting a route that can\'t get the +bundle to its final destination prior to the bundle's expiration time. + +#### **OWLT margin** + +One-way light time (OWLT) -- that is, distance -- is obviously a factor +in delivering a bundle to a node prior to a given time. OWLT can +actually change during the time a bundle is en route, but route +computation becomes intractably complex if we can\'t assume an OWLT +\"safety margin\" -- a maximum delta by which OWLT between any pair of +nodes can change during the time a bundle is in transit between them. + +We assume that the maximum rate of change in distance between any two +nodes in the network is about 150,000 miles per hour, which is about 40 +miles per second. (This was the speed of the Helios spacecraft, the +fastest man-made object launched to date.) + +At this speed, the distance between any two nodes that are initially +separated by a distance of N light seconds will increase by a maximum of +80 miles per second of transit (in the event that they are moving in +opposite directions). This will result in data arrival no later than +roughly (N + 2Q) seconds after transmission -- where the "OWLT margin" +value Q is (40 \* N) divided by 186,000 -- rather than just N seconds +after transmission as would be the case if the two nodes were stationary +relative to each other. When computing the expected time of arrival of a +transmitted bundle we simply use N + 2Q, the most pessimistic case, as +the anticipated total in-transit time. + +#### **Capacity** + +The *capacity* of a contact is the product of its data transmission rate +(in bytes per second) and its duration (stop time minus start time, in +seconds). + +#### **Estimated capacity consumption** + +The size of a bundle is the sum of its payload size and its header +size[^5], but bundle size is not the only lien on the capacity of a +contact. The total estimated volume consumption (or "EVC") for a bundle +is the sum of the sizes of the bundle's payload and header and the +estimated convergence-layer overhead. For a bundle whose header is of +size M and whose payload is of size N, the estimated convergence-layer +overhead is defined as 3% of (M+N), or 100 bytes, whichever is larger. + +#### **Residual capacity** + +The *residual capacity* of a given contact between the local node and +one of its neighbors, as computed for a given bundle, is the sum of the +capacities of that contact and all prior scheduled contacts between the +local node and that neighbor, less the sum of the ECCs of all bundles +with priority equal to or higher than the priority of the subject bundle +that are currently queued on the outduct for transmission to that +neighbor. + +#### **Excluded neighbors** + +A neighboring node C that refuses custody of a bundle destined for some +remote node D is termed an *excluded neighbor* for (that is, with +respect to computing routes to) D. So long as C remains an excluded +neighbor for D, no bundles destined for D will be forwarded to C -- +except that occasionally (once per lapse of the RTT between the local +node and C) a custodial bundle destined for D will be forwarded to C as +a "probe bundle". C ceases to be an excluded neighbor for D as soon as +it accepts custody of a bundle destined for D. + +#### **Critical bundles** + +A Critical bundle is one that absolutely has got to reach its +destination and, moreover, has got to reach that destination as soon as +is physically possible[^6]. + +For an ordinary non-Critical bundle, the CGR dynamic route computation +algorithm uses the routing table to select a single neighboring node to +forward the bundle through. It is possible, though, that due to some +unforeseen delay the selected neighbor may prove to be a sub-optimal +forwarder: the bundle might arrive later than it would have if another +neighbor had been selected, or it might not even arrive at all. + +For Critical bundles, the CGR dynamic route computation algorithm causes +the bundle to be inserted into the outbound transmission queues for +transmission to all neighboring nodes that can plausibly +forward the bundle to its final destination. The bundle is therefore +guaranteed to travel over the most successful route, as well as over all +other plausible routes. Note that this may result in multiple copies of +a Critical bundle arriving at the final destination. + +#### **Dynamic Route Selection Algorithm** + +Given a bundle whose destination is node D, we proceed as follows. + +First, if no contacts in the contact plan identify transmission to node +D, then we cannot use CGR to find a route for this bundle; CGR route +selection is abandoned. + +Next, if the contact plan has been modified in any way since routes were +computed for any nodes, we discard all routes for all nodes and +authorize route recomputation. (The contact plan changes may have +invalidated any or all of those earlier computations.) + +We create an empty list of Proximate Nodes (network neighbors) to send +the bundle to. + +We create a list of Excluded Nodes, i.e., nodes through which we will +not compute a route for this bundle. The list of Excluded +Nodes is initially populated with: + +- the node from which the bundle was directly received (so that we + avoid cycling the bundle between that node and the local node) -- + unless the Dynamic Route Selection Algorithm is being + re-applied due to custody refusal as discussed later; +- all excluded neighbors for the bundle's final destination node. + +If all routes computed for node D have been discarded due to contact +plan modification, then we must compute a new list of all routes from +the local node to D. To do so: + +- We construct an abstract contact graph, a directed acyclic graph + whose root is a notional contact from the local node to itself and + whose other vertices are all other contacts representing + transmission "from" some node such that a contact "to" that node + already exists in the graph, excluding contacts representing + transmission "to" some node such that a contact "from" that node + already exists in the graph. A terminal vertex is also included in + the graph, constituting a notional contact from node D to itself. +- We perform several Dijkstra searches within this graph, one search + for each of the local node's neighbors. On each search we find the + lowest-cost route that begins at the root of the graph and ends at + the terminal vertex. Each time a route is computed, we add it to the + list of routes for that route's entry node and then remove from + further consideration all contacts from the local node to the entry + node of that route. + + - The lowest-cost route computed during a search is the one that + is found to have the earliest best-case delivery time, where the + best-case delivery time characterizing a route is given by the + time at which a bundle would arrive at node D if transmitted at + the earliest possible moment of the last contact in the route + prior to the terminal vertex. + - Any contact whose end time is before the earliest possible time + that the bundle could arrive at the contact's sending node is + ignored. + - The earliest possible arrival time for the bundle on a given + contact is pessimistically computed as the sum of the bundle's + earliest possible transmission time plus the range in light + seconds from the contact's sending node to its receiving node, + plus the applicable one-way light time margin. + - The earliest possible transmission time for the bundle on a + given contact is the start time of the contact or bundle's + earliest possible arrival time at the contact's sending node, + whichever is later. +- If node D's list of entry nodes (route lists) is still empty, then + we cannot use CGR to find a route for this bundle; CGR route + selection is abandoned. + +We next examine all of the routes that are currently computed for +transmission of bundles to node D. + +- Any route whose termination time is in the past is deleted from the + list, and all contacts in that route whose termination time is in + the past are also deleted. But we then run another Dijkstra search + to compute the best route through the affected entry node given the + remaining contacts; if this search finds a route, the new route is + inserted into the appropriate location in the list. +- Any route whose best-case final delivery time is after the bundle's + expiration time is ignored, as is any route whose entry node is in + the list of Excluded Nodes. Loopback routes are also ignored unless + the local node is the bundle's final destination. +- For each route, the aggregate radiation time for this bundle on this + route is computed by summing the product of payload size and contact + transmission rate over all contacts in the route. Any route for + which the sum of best-case delivery time and aggregate radiation + time is after the bundle's expiration time is ignored. + +For each route that is not ignored, the route's entry node is added to +the list of Proximate Nodes for this bundle. Associated with the entry +node number in this list entry are the best-case final delivery time of +the route, the total number of "hops" in the route's end-to-end path, +and the forfeit time for transmission to this node. Forfeit time is the +route's termination time, the time by which the bundle must have been +transmitted to this node in order to have any chance of being forwarded +on this route. + +If, at the end of this procedure, the Proximate Nodes list is empty, +then we have been unable to use CGR to find a route for this bundle; CGR +route selection is abandoned. + +Otherwise: + +- If the bundle is flagged as a critical bundle, then a cloned copy of + this bundle is enqueued for transmission to every node in the + Proximate Nodes list. +- Otherwise, the bundle is enqueued for transmission on the outduct to + the most preferred neighbor in the Proximate Nodes list: + + - If one of the nodes in this list is associated with a best-case + delivery time that is earlier than that of all other nodes in + the list, then it is the most preferred neighbor. + - Otherwise, if one of the nodes with the earliest best-case + delivery time is associated with a smaller hop count than every + other node with the same best-case delivery time, then it is the + most preferred neighbor. + - Otherwise, the node with the smallest node number among all + nodes with the earliest best-case delivery time and smallest hop + count is arbitrarily chosen as the most preferred neighbor. + +### **Exception Handling** + +Conveyance of a bundle from source to destination through a DTN can fail +in a number of ways, many of which are best addressed by means of the +Delivery Assurance mechanisms described earlier. Failures in Contact +Graph Routing, specifically, occur when the expectations on which +routing decisions are based prove to be false. These failures of +information fall into two general categories: contact failure and +custody refusal. + +#### Contact Failure + +A scheduled contact between some node and its neighbor on the end-to-end +route may be initiated later than the originally scheduled start time, +or be terminated earlier than the originally scheduled stop time, or be +canceled altogether. Alternatively, the available capacity for a contact +might be overestimated due to, for example, diminished link quality +resulting in unexpectedly heavy retransmission at the convergence layer. +In each of these cases, the anticipated transmission of a given bundle +during the affected contact may not occur as planned: the bundle might +expire before the contact's start time, or the contact's stop time might +be reached before the bundle has been transmitted. + +For a non-Critical bundle, we handle this sort of failure by means of a +timeout: if the bundle is not transmitted prior to the forfeit time for +the selected Proximate Node, then the bundle is removed from its +outbound transmission queue and the Dynamic Route Computation Algorithm +is re-applied to the bundle so that an alternate route can be computed. + +#### Custody refusal + +A node that receives a bundle may find it impossible to forward it, for +any of several reasons: it may not have enough storage capacity to hold +the bundle, it may be unable to compute a forward route (static, +dynamic, or default) for the bundle, etc. Such bundles are simply +discarded, but discarding any such bundle that is marked for custody +transfer will cause a custody refusal signal to be returned to the +bundle's current custodian. + +When the affected bundle is non-Critical, the node that receives the +custody refusal re-applies the Dynamic Route Computation Algorithm to +the bundle so that an alternate route can be computed -- except that in +this event the node from which the bundle was originally directly +received is omitted from the initial list of Excluded Nodes. This +enables a bundle that has reached a dead end in the routing tree to be +sent back to a point at which an altogether different branch may be +selected. + +For a Critical bundle no mitigation of either sort of failure is +required or indeed possible: the bundle has already been queued for +transmission on all plausible routes, so no mechanism that entails +re-application of CGR's Dynamic Route Computation Algorithm could +improve its prospects for successful delivery to the final destination. +However, in some environments it may be advisable to re-apply the +Dynamic Route Computation Algorithm to all Critical bundles that are +still in local custody whenever a new Contact is added to the contact +graph: the new contact may open an additional forwarding opportunity for +one or more of those bundles. + +### **Remarks** + +The CGR routing procedures respond dynamically to the changes in network +topology that the nodes are able know about, i.e., those changes that +are subject to mission operations control and are known in advance +rather than discovered in real time. This dynamic responsiveness in +route computation should be significantly more effective and less +expensive than static routing, increasing total data return while at the +same time reducing mission operations cost and risk. + +Note that the non-Critical forwarding load across multiple parallel +paths should be balanced automatically: + +- Initially all traffic will be forwarded to the node(s) on what is + computed to be the best path from source to destination. +- At some point, however, a node on that preferred path may have so + much outbound traffic queued up that no contacts scheduled within + bundles' lifetimes have any residual capacity. This can cause + forwarding to fail, resulting in custody refusal. +- Custody refusal causes the refusing node to be temporarily added to + the current custodian's excluded neighbors list for the affected + final destination node. If the refusing node is the only one on the + path to the destination, then the custodian may end up sending the + bundle back to its upstream neighbor. Moreover, that custodian node + too may begin refusing custody of bundles subsequently sent to it, + since it can no longer compute a forwarding path. +- The upstream propagation of custody refusals directs bundles over + alternate paths that would otherwise be considered suboptimal, + balancing the queuing load across the parallel paths. +- Eventually, transmission and/or bundle expiration at the + oversubscribed node relieves queue pressure at that node and enables + acceptance of custody of a "probe" bundle from the upstream node. + This eventually returns the routing fabric to its original + configuration. + +Although the route computation procedures are relatively complex they +are not computationally difficult. The impact on computation resources +at the vehicles should be modest. + +### LTP Timeout Intervals + +Suppose we've got Earth ground station ES that is currently in view of +Mars but will be rotating out of view ("Mars-set") at some time T1 and +rotating back into view ("Mars-rise") at time T3. Suppose we've also got +Mars orbiter MS that is currently out of the shadow of Mars but will +move behind Mars at time T2, emerging at time T4. Let's also suppose +that ES and MS are 4 light-minutes apart (Mars is at its closest +approach to Earth). Finally, for simplicity, let's suppose that both ES +and MS want to be communicating at every possible moment (maximum link +utilization) but never want to waste any electricity. + +Neither ES nor MS wants to be wasting power on either transmitting or +receiving at a time when either Earth or Mars will block the signal. + +ES will therefore stop transmitting at either T1 or (T2 - 4 minutes), +whichever is earlier; call this time Tet0. It will stop receiving -- +that is, power off the receiver -- at either T1 or (T2 + 4 minutes), +whichever is earlier; call this time Ter0. It will resume transmitting +at either T3 or (T4 - 4 minutes), whichever is **late**, and +it will resume reception at either T3 or (T4 + 4 minutes), whichever is +later; call these times Tet1 and Ter1. + +Similarly, MS will stop transmitting at either T2 or (T1 - 4 minutes), +whichever is earlier; call this time Tmt0. It will stop receiving -- +that is, power off the receiver -- at either T2 or (T1 + 4 minutes), +whichever is earlier; call this time Tmr0. It will resume transmitting +at either T4 or (T3 - 4 minutes), whichever is later, and it will resume +reception at either T4 or (T3 + 4 minutes), whichever is later; call +these times Tmt1 and Tmr1. + +By making sure that we don't transmit when the signal would be blocked, +we guarantee that anything that is transmitted will arrive at a time +when it can be received. Any reception failure is due to data corruption +en route. + +So the moment of transmission of an acknowledgment to any message is +always equal to the moment the original message was sent plus some +imputed outbound queuing delay QO1 at the sending node, plus 4 minutes, +plus some imputed inbound and outbound queuing delay QI1 + QO2 at the +receiving node. The nominally expected moment of reception of this +acknowledgment is that moment of transmission plus 4 minutes, plus some +imputed inbound queuing delay QI2 at the original sending node. That is, +the timeout interval is 8 minutes + QO1 + QI1 + QO2 + QO2 -- *unless* +this moment of acknowledgement transmission is during an interval when +the receiving node is not transmitting, for whatever reason. In this +latter case, we want to suspend the acknowledgment timer during any +interval in which we know the remote node will not be transmitting. More +precisely, we want to add to the timeout interval the time difference +between the moment of message arrival and the earliest moment at which +the acknowledgment could be sent, i.e., the moment at which transmission +is resumed[^7]. + +So the timeout interval Z computed at ES for a message sent to MS at +time TX is given by: + +```c +Z = QO1 + 8 + QI1 + ((TA = TX + 4) > Tmt0 && TA < Tmt1) ? +Tmt1 - TA: 0) + QI2 + QO2 +``` + +This can actually be computed in advance (at time TX) if T1, T2, T3, +and T4 are known and are exposed to the protocol engine. + +If they are not exposed, then Z must initially be estimated to be (2 \* +the one-way light time) + QI + QO. The timer for Z must be dynamically +suspended at time Tmt0 in response to a state change as noted by +**ltpclock**. Finally, the timer must be resumed at time Tmt1 (in +response to another state change as noted by **ltpclock**), at which +moment the correct value for Z can be computed. + +### CFDP + +The ION implementation of CFDP is very simple, because only Class-1 +(Unacknowledged) functionality is implemented: the store-and-forward +routing performed by Bundle Protocol makes the CFDP Extended Procedures +unnecessary and the inter-node reliability provided by the CL protocol +underneath BP -- in particular, by LTP -- makes the CFDP Acknowledged +Procedures unnecessary. All that CFDP is required to do is segment and +reassemble files, interact with the underlying Unitdata Transfer layer +-- BP/LTP -- to effect the transmission and reception of file data +segments, and handle CFDP metadata including filestore requests. +CFDP-ION does all this, including support for cancellation of a file +transfer transaction by cancellation of the transmission of the bundles +encapsulating the transaction's protocol data units. + +Note that all CFDP data transmission is "by reference", via the ZCO +system, rather than "by value": the retransmission buffer for a bundle +containing CFDP file data is an extent of the original file itself, not +a copy retained in the ION database, and data received in bundles +containing CFDP PDU is written immediately to the appropriate location +in the reconstituted file rather than stored in the ION database. This +minimizes the space needed for the database. In general, file +transmission via CFDP is the most memory-efficient way to use ION in +flight operations. + +**Figure 15 A CFDP-ION entity** + +![](./media-ion-guide/media/image17.png) + +### **List data structures (lyst, sdrlist, smlist)** + +**Figure 16 ION list data structures** + +![](./media-ion-guide/media/image16.png) + +### **PSM Partition Structure** + +**Figure 17 psm partition structure** + +![](./media-ion-guide/media/image12.png) + +### **PSM and SDR Block Structures** + +**Figure 18 psm and sdr block structures** + +![](./media-ion-guide/media/image11.png) + +### **SDR Heap Structure** + +**Figure 19 sdr heap structure** + +![](./media-ion-guide/media/image19.png) + +## **Operation** + +The ION source distribution contains a README.TXT file with details on +building ION from source. For installations starts with the +open source distribution ION-DTN, using the standard sequence of + +- ./configure + + - Note: the user needs to clear all errors reported by the + configure script before proceeding. The distribution contains a + default "Makefile" at the top level of the distribution for + developer use. The configure script must complete successfully + to produce an updated Makefile. +- make +- sudo make install + +will build ION and install it under **/usr/local**. + +Users building from a clone of the repository need to use the command + +- autoreconf -fi + +before starting the installation. + +The "Build" instructions shown in the following sections for each +package are the instructions for building each package individually, for +ION development purposes. The default installation target for the +individual package build commands is **/opt.** + +One compile-time option is applicable to all ION packages: the platform +selection parameters --DVXWORKS and --DRTEMS affect the manner in which +most task instantiation functions are compiled. For VXWORKS and RTEMS, +these functions are compiled as library functions that must be +identified by name in the platform's symbol table, while for Unix-like +platforms they are compiled as main()functions. + +### Interplanetary Communication Infrastructure (ICI) + +#### **Compile-time options** + +Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DFSWSOURCE or +--DSM_SEMBASEKEY=0xff13), will alter the functionality of ION as noted +below. + +`PRIVATE_SYMTAB` + +This option causes ION to be built for VxWorks 5.4 or RTEMS with +reliance on a small private local symbol table that is accessed by means +of a function named sm_FindFunction. Both the table and the function +definition are, by default, provided by the symtab.c source file, which +is automatically included within the platform_sm.c source when this +option is set. The table provides the address of the top-level function +to be executed when a task for the indicated symbol (name) is to be +spawned, together with the priority at which that task is to execute and +the amount of stack space to be allocated to that task. + +PRIVATE_SYMTAB is defined by default for RTEMS but not for VxWorks 5.4. + +Absent this option, ION on VxWorks 5.4 must successfully execute the +VxWorks symFindByName function in order to spawn a new task. For this +purpose the entire VxWorks symbol table for the compiled image must be +included in the image, and task priority and stack space allocation must +be explicitly specified when tasks are spawned. + +`FSWLOGGER` + +This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file fswlogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters. + +`FSWCLOCK` + +This option causes the invocation of the standard time function within +getUTCTime (in ion.c) to be replaced (by #include) with code in the +source file fswutc.c, which might for example invoke a mission-specific +function to read a value from the spacecraft clock. A file of this name +must be in the inclusion path for the compiler. + +`FSWWDNAME` + +This option causes the invocation of the standard getcwd function within +cfdpInit (in libcfdpP.c) to be replaced (by #include) with code in the +source file wdname.c, which must in some way cause the mission-specific +value of the current working directory name to be copied into +cfdpdbBuf.workingDirectoryName. A file of this name must be in the +inclusion path for the compiler. + +`FSWSYMTAB` + +If the PRIVATE_SYMTAB option is also set, then the FSWSYMTAB option +causes the code in source file mysymtab.c to be included in +platform_sm.c in place of the default symbol table access implementation +in symtab.c. A file named mysymtab.c must be in the inclusion path for +the compiler. + +`FSWSOURCE` + +This option simply causes FSWLOGGER, FSWCLOCK, FSWWDNAME, and FSWSYMTAB +all to be set. + +`GDSLOGGER` + +This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file gdslogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters. + +GDSSOURCE + +This option simply causes GDSLOGGER to be set. + +`ION_OPS_ALLOC=*xx*` + +This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved for protocol operational state +information, i.e., is not available for the storage of bundles or LTP +segments. The default value is 20. + +`ION_SDR_MARGIN=*xx*` + +This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved simply as margin, for +contingency use. The default value is 20. + +The sum of ION_OPS_ALLOC and ION_SDR_MARGIN defines the amount of +non-volatile storage space that is sequestered at the time ION +operations are initiated: for purposes of congestion forecasting and +prevention of resource oversubscription, this sum is subtracted from the +total size of the SDR "heap" to determine the maximum volume of space +available for bundles and LTP segments. Data reception and origination +activities fail whenever they would cause the total amount of data store +space occupied by bundles and segments to exceed this limit. + +`USING_SDR_POINTERS` + +This is an optimization option for the SDR non-volatile data management +system: when set, it enables the value of any variable in the SDR data +store to be accessed directly by means of a pointer into the dynamic +memory that is used as the data store storage medium, rather than by +reading the variable into a location in local stack memory. Note that +this option must **not** be enabled if the data store is configured for +file storage only, i.e., if the SDR_IN_DRAM flag was set to zero at the +time the data store was created by calling sdr_load_profile. See the +ionconfig(5) man page in Appendix A for more information. + +`NO_SDR_TRACE` + +This option causes non-volatile storage utilization tracing functions to +be omitted from ION when the SDR system is built. It disables a useful +debugging option but reduces the size of the executable software. + +`NO_PSM_TRACE` + +This option causes memory utilization tracing functions to be omitted +from ION when the PSM system is built. It disables a useful debugging +option but reduces the size of the executable software. + +`IN_FLIGHT` + +This option controls the behavior of ION when an unrecoverable error is +encountered. + +If it is set, then the status message "Unrecoverable SDR error" is +logged and the SDR non-volatile storage management system is globally +disabled: the current database access transaction is ended and (provided +transaction reversibility is enabled) rolled back, and all ION tasks +terminate. + +Otherwise, the ION task that encountered the error is simply aborted, +causing a core dump to be produced to support debugging. + +`SM_SEMKEY=0x*XXXX*` + +This option overrides the default value (0xee01) of the identifying +"key" used in creating and locating the global ION shared-memory system +mutex. + +`SVR4_SHM` + +This option causes ION to be built using svr4 shared memory as the +pervasive shared-memory management mechanism. svr4 shared memory is +selected by default when ION is built for any platform other than MinGW, +VxWorks 5.4, or RTEMS. (For these latter operating systems all memory is +shared anyway, due to the absence of a protected-memory mode.) + +`POSIX1B_SEMAPHORES` + +This option causes ION to be built using POSIX semaphores as the +pervasive semaphore mechanism. POSIX semaphores are selected by default +when ION is built for RTEMS but are otherwise not used or supported; +this option enables the default to be overridden. + +`SVR4_SEMAPHORES` + +This option causes ION to be built using svr4 semaphores as the +pervasive semaphore mechanism. svr4 semaphores are selected by default +when ION is built for any platform other than MinGW (for which Windows +event objects are used), VxWorks 5.4 (for which VxWorks native +semaphores are the default choice), or RTEMS (for which POSIX semaphores +are the default choice). + +`SM_SEMBASEKEY=0x*XXXX*` + +This option overrides the default value (0xee02) of the identifying +"key" used in creating and locating the global ION shared-memory +semaphore database, in the event that svr4 semaphores are used. + +`SEMMNI=*xxx*` + +This option declares to ION the total number of svr4 semaphore sets +provided by the operating system, in the event that svr4 semaphores are +used. It overrides the default value, which is 10 for Cygwin and 128 +otherwise. (Changing this value typically entails rebuilding the O/S +kernel.) + +`SEMMSL=*xxx*` + +This option declares to ION the maximum number of semaphores in each +svr4 semaphore set, in the event that svr4 semaphores are used. It +overrides the default value, which is 6 for Cygwin and 250 otherwise. +(Changing this value typically entails rebuilding the O/S kernel.) + +`SEMMNS=*xxx*` + +This option declares to ION the total number of svr4 semaphores that the +operating system can support; the maximum possible value is SEMMNI x +SEMMSL. It overrides the default value, which is 60 for Cygwin and 32000 +otherwise. (Changing this value typically entails rebuilding the O/S +kernel.) + +`ION_NO_DNS` + +This option causes the implementation of a number of Internet socket I/O +operations to be omitted for ION. This prevents ION software from being +able to operate over Internet connections, but it prevents link errors +when ION is loaded on a spacecraft where the operating system does not +include support for these functions. + +`ERRMSGS_BUFSIZE=*xxxx*` + +This option set the size of the buffer in which ION status messages are +constructed prior to logging. The default value is 4 KB. + +`SPACE_ORDER=*x*` + +This option declares the word size of the computer on which the compiled +ION software will be running: it is the base-2 log of the number of +bytes in an address. The default value is 2, i.e., the size of an +address is 2^2^ = 4 bytes. For a 64-bit machine, SPACE_ORDER must be +declared to be 3, i.e., the size of an address is 2^3^ = 8 bytes. + +`NO_SDRMGT` + +This option enables the SDR system to be used as a data access +transaction system only, without doing any dynamic management of +non-volatile data. With the NO_SDRMGT option set, the SDR system library +can (and in fact must) be built from the sdrxn.c source file alone. + +`DOS_PATH_DELIMITER` + +This option causes ION_PATH_DELIMITER to be set to '\\' (backslash), for +use in construction path names. The default value of ION_PATH_DELIMITER +is '/' (forward slash, as is used in Unix-like operating systems). + +#### **Build** + +To build ICI for a given deployment platform: + +1. Decide where you want ION's executables, libraries, header files, + etc. to be installed. The ION makefiles all install their build + products to subdirectories (named **bin**, **lib**, **include**, + **man**, **man/man1**, **man/man3**, **man/man5**) of an ION root + directory, which by default is the directory named **/opt**. If you + wish to use the default build configuration, be sure that the + default directories (**/opt/bin**, etc.) exist; if not, select + another ION root directory name -- this document will refer to it as + **\$OPT** -- and create the subdirectories as needed. In any case, + make sure that you have read, write, and execute permission for all + of the ION installation directories and that: + + - The directory **/\$OPT/bin** is in your execution path. + - The directory **/\$OPT/lib** is in your \$LD_LOADLIB_PATH. +2. Edit the Makefile in **ion/ici**: + +- Make sure PLATFORMS is set to the appropriate platform name, e.g., + x86-redhat, sparc-sol9, etc. +- Set OPT to the directory where you want to install the ici packages + you build, if other than "/opt" (for example: /usr/local). + +3. Then: + +```bash +cd ion/ici +sudo make +sudo make install +``` + +#### **Configure** + +Three types of files are used to provide the information needed to +perform global configuration of the ION protocol stack: the ION system +configuration (or **ionconfig**) file, the ION administration command +(**ionrc**) file, and the ION security configuration (**ionsecrc**) +file. For details, see the man pages for ionconfig(5), ionrc(5), and +ionsecrc(5) in Appendix A. + +Normally the instantiation of ION on a given computer establishes a +single ION node on that computer, for which hard-coded values of wmKey +and sdrName (see ionconfig(5)) are used in common by all executables to +assure that all elements of the system operate within the same state +space. For some purposes, however, it may be desirable to establish +multiple ION nodes on a single workstation. (For example, constructing +an entire self-contained DTN network on a single machine may simplify +some kinds of regression testing.) ION supports this configuration +option as follows: + +- Multi-node operation on a given computer is enabled if and only if + the environment variable ION_NODE_LIST_DIR is defined in the + environment of every participating ION process. Moreover, the value + assigned to this variable must be the same text string in the + environments of all participating ION processes. That value must be + the name (preferably, fully qualified) of the directory in which the + ION multi-node database file "ion_nodes" will reside. +- The definition of ION_NODE_LIST_DIR makes it possible to establish + up to one ION node per directory rather than just one ION node on + the computer. When **ionadmin** is used to establish a node, the + ionInitialize() function will get that node's wmKey and sdrName from + the .ionconfig file, use them to allocate working memory and create + the SDR database, and then write a line to the ion_nodes file noting + the nodeNbr, wmKey, sdrName, and wdName for the node it just + initialized. wdName is the current working directory in which + **ionadmin** was running at the time ionInitialize()is called; it is + the directory within which the node resides. +- This makes it easy to connect all the node\'s daemon processes -- + running within the same current working directory -- to the correct + working memory partition and SDR database: the ionAttach() function + simply searches the ion_nodes file for a line whose wdName matches + the current working directory of the process that is trying to + attach, then uses that line\'s wmKey and sdrName to link up. +- It is also possible to initiate a process from within a directory + other than the one in which the node resides. To do so, define the + additional environment variable ION_NODE_WDNAME in the shell from + which the new process is to be initiated. When ionAttach() is called + it will first try to get \"current working directory\" (for ION + attachment purposes **only**) from that environment variable; only + if ION_NODE_WDNAME is undefined will it use the actual cwd that it + gets from calling igetcwd(). +- When running multiple ION instances, the first ION instance must have the largest sdr working memory specified by `sdrWmSize` or default. If any later ION instance launched with `sdrWmSize` exceeding the first ION instance, it will result in crash upon launch. It is further recommended that all ION instances running simultaneously on a single host should set their `sdrWmSize` the same. + +#### **Run** + +The executable programs used in operation of the ici component of ION +include: + +- The **ionadmin** system configuration utility and **ionsecadmin** + security configuration utility, invoked at node startup time and as + needed thereafter. +- The **rfxclock** background daemon, which affects scheduled network + configuration events. +- The **sdrmend** system repair utility, invoked as needed. +- The **sdrwatch** and **psmwatch** utilities for resource utilization + monitoring, invoked as needed. + +Each time it is executed, **ionadmin** computes a new congestion +forecast and, if a congestion collapse is predicted, invokes the node's +congestion alarm script (if any). **ionadmin** also establishes the node +number for the local node and starts/stops the **rfxclock** task, among +other functions. For further details, see the man pages for ionadmin(1), +ionsecadmin(1), rfxclock(1), sdrmend(1), sdrwatch(1), and psmwatch(1) in +Appendix A. + +#### **Test** + +Six test executables are provided to support testing and debugging of +the ICI component of ION: + +- The **file2sdr** and **sdr2file** programs exercise the SDR system. +- The **psmshell** program exercises the PSM system. +- The **file2sm**, **sm2file**, and **smlistsh** programs exercise the + shared-memory linked list system. + +For details, see the man pages for file2sdr(1), sdr2file(1), +psmshell(1), file2sm(1), sm2file(1), and smlistsh(1) in Appendix A. + +### Licklider Transmission Protocol (LTP) + +#### **Build** + +To build LTP: + +1. Make sure that the "ici" component of ION has been built for the + platform on which you plan to run LTP. +2. Edit the Makefile in **ion/ltp**: + +- As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run LTP. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). + +3. Then: + +```bash +cd ion/ltp +make +sudo make install +``` + +#### **Configure** + +The LTP administration command (**ltprc**) file provides the information +needed to configure LTP on a given ION node. For details, see the man +page for ltprc(5) in Appendix A. + +#### **Run** + +The executable programs used in operation of the ltp component of ION +include: + +- The **ltpadmin** protocol configuration utility, invoked at node + startup time and as needed thereafter. +- The **ltpclock** background daemon, which affects scheduled LTP + events such as segment retransmissions. +- The **ltpmeter** block management daemon, which segments blocks and + effects LTP flow control. +- The **udplsi** and **udplso** link service input and output tasks, + which handle transmission of LTP segments encapsulated in UDP + datagrams (mainly for testing purposes). + +**ltpadmin** starts/stops the **ltpclock** and ltpmeter tasks and, as +mandated by configuration, the **udplsi** and **udplso** tasks. + +For details, see the man pages for ltpadmin(1), ltpclock(1), +ltpmeter(1), udplsi(1), and udplso(1) in Appendix A. + +#### **Test** + +Two test executables are provided to support testing and debugging of +the LTP component of ION: + +- **ltpdriver** is a continuous source of LTP segments. +- **ltpcounter** is an LTP block receiver that counts blocks as they + arrive. + +For details, see the man pages for ltpdriver(1) and ltpcounter(1) in +Appendix A. + +### Bundle Streaming Service Protocol (BSSP) + +#### **Build** + +To build BSSP: + +1. Make sure that the "ici" component of ION has been built for the + platform on which you plan to run BSSP. +2. Edit the Makefile in **ion/bssp**: + +- As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BSSP. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). + +3. Then: + +```bash +cd ion/bssp +make +sudo make install +``` + +#### **Configure** + +The BSSP administration command (**bssprc**) file provides the +information needed to configure BSSP on a given ION node. For details, +see the man page for bssprc(5) in Appendix A. + +The bssprc file has a command option specifying the max_block_size. This +is to prevent retransmission inefficiency when the blocks size of a +stream data is too large. The unit of retransmission for BSSP is the +block, so if the block size is too large, it is very expensive to the +network to provide retransmission. If one needs bulk data transfer, +instead of streaming, one should use BP with reliability LTP instead of +using BSSP. If you are using udpbso and udpbsi as the underlying +convergence layer, then the max_block_size parameter for bssprc cannot +be larger than 65507 bytes, because each UDP datagram can only be as +large as 65507 bytes (payload) + 20 (IP Header) + 8 (UDP Header) = 65535 +byte. + +#### **Run** + +The executable programs used in operation of the bssp component of ION +include: + +- The **bsspadmin** protocol configuration utility, invoked at node + startup time and as needed thereafter. +- The **bsspclock** background daemon, which affects scheduled BSSP + events such as segment retransmissions. +- The **udpbsi** and **udpbso** link service input and output tasks, + which handle transmission of BSSP segments encapsulated in UDP + datagrams (mainly for testing purposes; in space domain, the + appropriate CCSDS link layer will be used instead of UDP). + +**bsspadmin** starts/stops the **bsspclock** task and, as mandated by +configuration, the **udpbsi** and **udblso** tasks. + +For details, see the man pages for bsspadmin(1), bsspclock(1), +bsspmeter(1), udpbsi(1), and udpbso(1) in Appendix A. + +### Bundle Protocol (BP) + +#### **Compile-time options** + +Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DION_NOSTATS or +--DBRSTERM=60), will alter the functionality of BP as noted below. + +#### TargetFFS + +Setting this option adapts BP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a BP task +keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-BP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the BP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the BP task -- which does not need it -- at +the expense of normally higher-priority tasks, and can thereby introduce +runtime anomalies. BP tasks should therefore close files immediately +after each access when running on a VxWorks platform that uses the +TargetFFS flash file system. The TargetFFS compile-time option ensures +that they do so. + +#### BRSTERM=*xx* + +This option sets the maximum number of seconds by which the current time +at the BRS server may exceed the time tag in a BRS authentication +message from a client; if this interval is exceeded, the authentication +message is presumed to be a replay attack and is rejected. Small values +of BRSTERM are safer than large ones, but they require that clocks be +more closely synchronized. The default value is 5. + +#### ION_NOSTATS + +Setting this option prevents the logging of bundle processing statistics +in status messages. + +#### KEEPALIVE_PERIOD=*xx* + +This option sets the number of seconds between transmission of +keep-alive messages over any TCP or BRS convergence-layer protocol +connection. The default value is 15. + +#### ION_BANDWIDTH_RESERVED + +Setting this option overrides strict priority order in bundle +transmission, which is the default. Instead, bandwidth is shared between +the priority-1 and priority-0 queues on a 2:1 ratio whenever there is no +priority-2 traffic. + +#### ENABLE_BPACS + +This option causes Aggregate Custody Signaling source code to be +included in the build. ACS is alternative custody transfer signaling +mechanism that sharply reduces the volume of custody acknowledgment +traffic. + +#### ENABLE_IMC + +This option causes IPN Multicast source code to be included in the +build. IMC is discussed in section 1.8.4 above. + +#### **Build** + +To build BP: + +1. Make sure that the "ici", "ltp", "dgr", and "bssp" components of ION + have been built for the platform on which you plan to run BP. +2. Edit the Makefile in **ion/bp**: + +- As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BP. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). + +3. Then: + +```bash +cd ion/bp +make +sudo make install +``` + +#### **Configure** + +The BP administration command (**bprc**) file provides the information +needed to configure generic BP on a given ION node. The IPN scheme +administration command (**ipnrc**) file provides information that +configures static and default routes for endpoints whose IDs conform to +the "ipn" scheme. The DTN scheme administration command (**dtn2rc**) +file provides information that configures static and default routes for +endpoints whose IDs conform to the "dtn" scheme, as supported by the +DTN2 reference implementation. For details, see the man pages for +bprc(5), ipnrc(5), and dtn2rc(5) in Appendix A. + +#### **Run** + +The executable programs used in operation of the bp component of ION +include: + +- The **bpadmin, ipnadmin,** and **dtn2admin** protocol configuration + utilities, invoked at node startup time and as needed thereafter. +- The **bpclock** background daemon, which effects scheduled BP events + such as TTL expirations and which also implements rate control. +- The **ipnfw** and **dtn2fw** forwarding daemons, which compute + routes for bundles addressed to "ipn"-scheme and "dtn"-scheme + endpoints, respectively. +- The **ipnadminep** and **dtn2adminep** administrative endpoint + daemons, which handle custody acceptances, custody refusals, and + status messages. +- The **bpclm** background daemon, which selects convergence-layer + outducts by which bundles are transmitted to neighboring nodes. +- The **brsscla** (server) and **brsccla** (client) Bundle Relay + Service convergence-layer adapters. +- The **tcpcli** (input) TCP convergence-layer adapter, which includes + convergence-layer output functionality in privately managed threads. +- The s**tcpcli** (input) and s**tcpclo** (output) simplified TCP + convergence-layer adapters. +- The **udpcli** (input) and **udpclo** (output) UDP convergence-layer + adapters. +- The **ltpcli** (input) and **ltpclo** (output) LTP convergence-layer + adapters. +- The **dgrcla** Datagram Retransmission convergence-layer adapter. +- The **bpsendfile** utility, which sends a file of arbitrary size, + encapsulated in a single bundle, to a specified BP endpoint. +- The **bpstats** utility, which prints a snapshot of currently + accumulated BP processing statistics on the local node. +- The **bptrace** utility, which sends a bundle through the network to + enable a forwarding trace based on bundle status reports. +- The **lgsend** and **lgagent** utilities, which are used for remote + administration of ION nodes. +- The **hmackeys** utility, which can be used to create hash keys + suitable for use in bundle authentication blocks and BRS + convergence-layer protocol connections. + +**bpadmin** starts/stops the **bpclock** task and, as mandated by +configuration, the **ipnfw**, **dtn2fw**, **ipnadminep**, +**dtn2adminep**, bpclm, **brsscla**, **brsccla**, tcpcli, s**tcpcli**, +s**tcpclo**, **udpcli**, **udpclo**, **ltpcli**, **ltpclo**, and +**dgrcla** tasks. + +For details, see the man pages for bpadmin(1),ipnadmin(1), dtn2admin(1), +bpclock(1), bpclm(1), ipnfw(1), dtn2fw(1), ipnadminep(1), +dtn2adminep(1), brsscla(1), brsccla(1),tcpcli(1), stcpcli(1), +stcpclo(1), udpcli(1), udpclo(1), ltpcli(1), ltpclo(1), dgrcla(1), +bpsendfile(1), bpstats(1), bptrace(1), lgsend(1), lgagent(1), and +hmackeys(1) in Appendix A. + +#### **Test** + +Five test executables are provided to support testing and debugging of +the BP component of ION: + +- **bpdriver** is a continuous source of bundles. +- **bpcounter** is a bundle receiver that counts bundles as they + arrive. +- **bpecho** is a bundle receiver that sends an "echo" acknowledgment + bundle back to bpdriver upon reception of each bundle. +- **bpsource** is a simple console-like application for interactively + sending text strings in bundles to a specified DTN endpoint, + nominally a **bpsink** task. +- **bpsink** is a simple console-like application for receiving + bundles and printing their contents. + +For details, see the man pages for bpdriver(1), bpcounter(1), bpecho(1), +bpsource(1), and bpsink(1) in Appendix A. + +### Datagram Retransmission (DGR) + +#### **Build** + +To build DGR: + +1. Make sure that the "ici" component of ION has been built for the + platform on which you plan to run DGR. +2. Edit the Makefile in **ion/dgr**: + +- As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run DGR. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). + +3. Then: + +```bash +cd ion/dgr +make +sudo make install +``` + +#### **Configure** + +No additional configuration files are required for the operation of the +DGR component of ION. + +#### **Run** + +No runtime executables are required for the operation of the DGR +component of ION. + +#### **Test** + +Two test executables are provided to support testing and debugging of +the DGR component of ION: + +- **file2dgr** repeatedly reads a file of text lines and sends copies + of those text lines via DGR to **dgr2file**, which writes them to a + copy of the original file. + +For details, see the man pages for file2dgr(1) and dgr2file(1) in +Appendix A. + +### Asynchronous Message Service (AMS) + +#### **Compile-time options** + +Note that, by default, the syntax by which AMS MIB information is +presented to AMS is as documented in the "amsrc" man page. Alternatively +it is possible to use an XML-based syntax as documented in the "amsxml" +man page. To use the XML-based syntax instead, be sure that the "expat" +XML interpretation system is installed and pass the argument +"\--with-expat" to "./configure" when building ION. + +Defining the following macros, by setting parameters that are provided +to the C compiler (for example, DAMS_INDUSTRIAL), will alter the +functionality of AMS as noted below. + +`AMS_INDUSTRIAL` + +Setting this option adapts AMS to an "industrial" rather than +safety-critical model for memory management. By default, the memory +acquired for message transmission and reception buffers in AMS is +allocated from limited ION working memory, which is fixed at ION +start-up time; this limits the rate at which AMS messages may be +originated and acquired. When --DAMS_INDUSTRIAL is set at compile time, +the memory acquired for message transmission and reception buffers in +AMS is allocated from system memory, using the familiar malloc() and +free() functions; this enables much higher message traffic rates on +machines with abundant system memory. + +#### **Build** + +To build AMS: + +1. Make sure that the "bp" component of ION has been built for the + platform on which you plan to run AMS. +2. Edit the Makefile in **ion/cfdp**: + +- Just as for bp, make sure PLATFORMS is set to the name of the + platform on which you plan to run AMS. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). + +3. Then: + +```bash +cd ion/ams +make +sudo make install +``` + +#### **Configure** + +There is no central configuration of AMS; each AMS entity (configuration +server, registrar, or application module) is individually configured at +the time its initial MIB is loaded at startup. Note that a single MIB +may be shared between multiple AMS entities without issue. + +For details of MIB file syntax, see the man pages for amsrc(5) and +amsxml(5) in Appendix A. + +#### **Run** + +The executable programs used in operation of the AMS component of ION +include: + +- The **amsd** background daemon, which serves as configuration server + and/or as the registrar for a single application cell. +- The **ramsgate** application module, which serves as the Remote AMS + gateway for a single message space. +- The **amsstop** utility, which terminates all AMS operation + throughout a single message space. +- The **amsmib** utility, which announces supplementary MIB + information to selected subsets of AMS entities without interrupting + the operation of the message space. + +For details, see the man pages for amsd(1), ramsgate(1), amsstop(1), and +amsmib(1) in Appendix A. + +#### **Test** + +Seven test executables are provided to support testing and debugging of +the AMS component of ION: + +- **amsbenchs** is a continuous source of messages. +- **amsbenchr** is a message receiver that calculates bundle + transmission performance statistics. +- **amshello** is an extremely simple AMS "hello, world" demo program + -- a self-contained distributed application in a single source file + of about seventy lines. +- **amsshell** is a simple console-like application for interactively + publishing, sending, and announcing text strings in messages. +- **amslog** is a simple console-like application for receiving + messages and piping their contents to stdout. +- **amslogprt** is a pipeline program that simply prints AMS message + contents piped to it from amslog. +- **amspubsub** is a pair of functions for rudimentary testing of AMS + functionality in a VxWorks environment. + +For details, see the man pages for amsbenchs(1), amsbenchr(1), +amshello(1), amsshell(1), amslog(1), amslogprt(1), amspub(1), and +amssub(1) in Appendix A. + +For further operational details of the AMS system, please see sections 4 +and 5 of the [AMS Programmer's Guide.](./AMS-Programmer-Guide.md) + +### CCSDS File Delivery Protocol (CFDP) + +#### **Compile-time options** + +Defining the following macro, by setting a parameter that is provided to +the C compiler (i.e., --DTargetFFS), will alter the functionality of +CFDP as noted below. + +#### TargetFFS + +Setting this option adapts CFDP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a CFDP +task keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-CFDP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the CFDP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the CFDP task -- which does not need it -- +at the expense of normally higher-priority tasks, and can thereby +introduce runtime anomalies. CFDP tasks should therefore close files +immediately after each access when running on a VxWorks platform that +uses the TargetFFS flash file system. The TargetFFS compile-time option +ensures that they do so. + +#### **Build** + +To build CFDP: + +1. Make sure that the "bp" component of ION has been built for the + platform on which you plan to run CFDP. +2. Edit the Makefile in **ion/cfdp**: + +- Just as for bp, make sure PLATFORMS is set to the name of the + platform on which you plan to run CFDP. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed. + +3. Then: + +```bash +cd ion/cfdp + +make + +make install +``` + +#### **Configure** + +The CFDP administration command (**cfdprc**) file provides the +information needed to configure CFDP on a given ION node. For details, +see the man page for cfdprc(5) in Appendix A. + +#### **Run** + +The executable programs used in operation of the CFDP component of ION +include: + +- The **cfdpadmin** protocol configuration utility, invoked at node + startup time and as needed thereafter. +- The **cfdpclock** background daemon, which affects scheduled CFDP + events such as check timer expirations. The **cfdpclock** task also + affects CFDP transaction cancellations, by canceling the bundles + encapsulating the transaction's protocol data units. +- The **bputa** UT-layer input/output task, which handles transmission + of CFDP PDUs encapsulated in bundles. + +**cfdpadmin** starts/stops the **cfdpclock** task and, as mandated by +configuration, the **bputa** task. + +For details, see the man pages for cfdpadmin(1), cfdpclock(1), and +bputa(1) in Appendix A. + +#### **Test** + +A single executable, **cfdptest**, is provided to support testing and +debugging of the DGR component of ION. For details, see the man page for +cfdptest(1) in Appendix A. + +### Bundle Streaming Service (BSS) + +#### **Compile-time options** + +Defining the following macro, by setting a parameter that is provided to +the C compiler (e.g., --DWINDOW=10000), will alter the functionality of +BSS as noted below. + +#### WINDOW=*xx* + +Setting this option changes the maximum number of seconds by which the +BSS database for a BSS application may be "rewound" for replay. The +default value is 86400 seconds, which is 24 hours. + +#### **Build** + +To build BSS: + +- Make sure that the "bp" component of ION has been built for the + platform on which you plan to run BSS. +- Edit the Makefile in **ion/bss**: +- As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BSS. +- Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local). +- Then: + +```bash +cd ion/bss + +make + +sudo make install +``` + +#### **Configure** + +No additional configuration files are required for the operation of the +BSS component of ION. + +#### **Run** + +No runtime executables are required for the operation of the BSS +component of ION. + +#### **Test** + +Four test executables are provided to support testing and debugging of +the BSS component of ION: + +- **bssdriver** sends a stream of data to **bsscounter** for + non-interactive testing. +- **bssStreamingApp** sends a stream of data to **bssrecv** for + graphical, interactive testing. + +For details, see the man pages for bssdriver(1), bsscounter(1), +bssStreamingApp(1), and bssrecv(1) in Appendix A. + +[^1]: In ION, reliable convergence-layer protocols (where available) are + by default used for every bundle. The application can instead + mandate selection of "best-effort" service at the convergence layer + by setting the BP_BEST_EFFORT flag in the "extended class of service + flags" parameter, but this feature is an ION extension that is not + supported by other BP implementations at the time of this writing. + +[^2]: Note that, in all occupancy figures, ION data management accounts + not only for the sizes of the payloads of all queued bundles but + also for the sizes of their headers. + +[^3]: Barring data loss or corruption for which the various + retransmission mechanisms in ION cannot compensate. + +[^4]: Note that ION may indeed block the offering of a message to the + network, but this is local admission control -- assuring that the + node's local buffer space for queuing outbound bundles is not + oversubscribed -- rather than end-to-end flow control. It is always + possible for there to be ample local buffer space yet insufficient + network capacity to convey the offered data to their final + destination, and vice versa. + +[^5]: The minimum size of an ION bundle header is 26 bytes. Adding + extension blocks (such as those that effect the Bundle Security + Protocol) will increase this figure. + +[^6]: In ION, all bundles are by default non-critical. The application + can indicate that data should be sent in a Critical bundle by + setting the BP_MINIMUM_LATENCY flag in the "extended class of + service" parameter, but this feature is an ION extension that is not + supported by other BP implementations at the time of this writing. + +[^7]: If we wanted to be extremely accurate we could also + subtract from the timeout interval the imputed inbound + queuing delay QI, since inbound queuing would presumably be + completed during the interval in which transmission was suspended. + But since we're guessing at the queuing delays anyway, this + adjustment doesn't make a lot of sense. +# Interplanetary Overlay Network (ION) Design and Operation's Guide + +**Version 4.1.3** +**JPL D-48259** + +**Document Change Log** + +| Ver No. | Date | Description | Note | +| ------- | ----------- | ----------------------------------------------------------------------- | --------------- | +| V4.1.3 | 12/08/2023 | converted to MarkDown | | +| V4.0.1 | 11/20/2020 | ION 4.0.1 | | +| V3.6.2 | 11/19/2018 | ION 3.6.2 release features | Skipped V3.6.1. | +| V3.6 | 12/31/2017 | ION 3.6 release features | Skipped V3.5. | +| V3.4 | 3/28/2016 | ION 3.4 release features | | +| V3.3 | 3/4/2015 | ION 3.3 release features | | +| V3.2 | 12/17/2013 | ION 3.2 release features | | +| V3.1 | 9/28/2012 | ION 3.1 release features | | +| V3.0 | 3/22/2012 | Align with ION 3.0 release | | +| V1.13 | 10/13/2011 | Updates for Source Forge Release | | +| V1.12 | 6/11/2010 | Updates for second open source release (2.2) | | +| V1.11 | 12/11/2009 | BRS updates, multi-node config | | +| V1.10 | 10/23/2009 | Final additions prior to DINET 2 experiment | | +| V1.9 | 6/29/2009 | Add updates for DINET 2, including CFDP, ionsec | | +| V1.8 | 2/6/2009 | Update discussion of Contact Graph Routing; document status msg formats | | +| V1.7 | 12/1/2008 | Add documentation for OWLT simulator, BP extension | | +| V1.6 | 10/03/2008 | Add documentation of sm_SemUnend | | +| V1.5 | 09/20/2008 | Revisions requested SQA | | +| V1.4 | 07/31/2008 | Add a section on optimizing ION-based network; tuning | | +| V1.3 | 07/08/2008 | Revised some details of CGR | | +| V1.2 | 05/24/2008 | Revised man pages for bptrace, ltprc, bprc. | | +| V1.1 | 05/18/2008 | Some additional diagrams | | +| V1.0 | 04/28/2008 | Initial version of ION design and ops manual | | + +## Design + +The Interplanetary Overlay Network (ION) software distribution is an +implementation of Delay-Tolerant Networking (DTN) architecture as +described in Internet RFC 4838. It is designed to enable inexpensive +insertion of DTN functionality into embedded systems such as robotic +spacecraft. The intent of ION deployment in space flight mission systems +is to reduce cost and risk in mission communications by simplifying the +construction and operation of automated digital data communication +networks spanning space links, planetary surface links, and terrestrial +links. + +A comprehensive overview of DTN is beyond the scope of this document. +Very briefly, though, DTN is a digital communication networking +technology that enables data to be conveyed between two communicating +entities automatically and reliably even if one or more of the network +links in the end-to-end path between those entities is subject to very +long signal propagation latency and/or prolonged intervals of +unavailability. + +The DTN architecture is much like the architecture of the Internet, +except that it is one layer higher in the familiar ISO protocol "stack". +The DTN analog to the Internet Protocol (IP), called "Bundle Protocol" +(BP), is designed to function as an "overlay" network protocol that +interconnects "internets" -- including both Internet-structured networks +and also data paths that utilize only space communication links as +defined by the Consultative Committee for Space Data Systems (CCSDS) -- +in much the same way that IP interconnects "subnets" such as those built +on Ethernet, SONET, etc. By implementing the DTN architecture, ION +provides communication software configured as a protocol stack that +looks like this: + +![Figure 1](./media-ion-guide/media/image2.png) + +**Figure 1 DTN protocol stack** + +Data traversing a DTN are conveyed in DTN *bundles* -- which are +functionally analogous to IP packets -- between BP *endpoints* which are +functionally analogous to sockets. Multiple BP endpoints may be accessed +at a single DTN *node* -- functionally analogous to a network interface +card -- and multiple nodes may reside on the same computer just as a +single computer (host or router) in the Internet may have multiple +network interface cards. + +BP endpoints are identified by Universal Record Identifiers (URIs), +which are ASCII text strings of the general form: + +*scheme_name*:*scheme_specific_part* + +For example: + +dtn://topquark.caltech.edu/mail + +But for space flight communications this general textual representation +might impose more transmission overhead than missions can afford. For +this reason, ION is optimized for networks of endpoints whose IDs +conform more narrowly to the following scheme: + +ipn:*node_number.service_number* + +This enables them to be abbreviated to pairs of unsigned binary integers +via a technique called Compressed Bundle Header Encoding (CBHE). +CBHE-conformant BP *endpoint IDs* (EIDs) are not only functionally +similar to Internet socket addresses but also structurally similar: node +numbers are roughly analogous to Internet node numbers (IP addresses), +in that they typically identify the flight or ground data system +computers on which network software executes, and service numbers are +roughly analogous to TCP and UDP port numbers. + +More generally, the node numbers in CBHE-conformant BP endpoint IDs are +one manifestation of the fundamental ION notion of *network* *node +number*: in the ION architecture there is a natural one-to-one mapping +not only between node numbers and BP endpoint node numbers but also +between node numbers and: + +- LTP engine IDs +- AMS continuum numbers +- CFDP entity numbers + +Starting with version 3.1 of ION, this endpoint naming rule is +experimentally extended to accommodate *bundle multicast*, i.e., the +delivery of copies of a single transmitted bundle to multiple nodes at +which interest in that bundle's payload has been expressed. Multicast in +ION -- "Interplanetary Multicast" (IMC) -- is accomplished by simply +issuing a bundle whose destination endpoint ID conforms to the following +scheme: + +imc:*group_number.service_number* + +A copy of the bundle will automatically be delivered at every node that +has registered in the destination endpoint. + +(Note: for now, the operational significance of a given group number +must be privately negotiated among ION users. If this multicast +mechanism proves useful, IANA may at some point establish a registry for +IMC group numbers. **Also note that a new mechanism for bundle multicast +is introduced in ION 4.0.1, along with support for Bundle Protocol +version 7.** This new mechanism vastly simplifies bundle multicast; +chiefly, the **imcadmin** utility is deprecated.) + +### Structure and function + +The ION distribution comprises the following software packages: + +- ici (Interplanetary Communication Infrastructure), a set of + general-purpose libraries providing common functionality to the + other packages. The ici package includes a security policy component + that supports the implementation of security mechanisms at multiple + layers of the protocol stack. +- ltp (Licklider Transmission Protocol), a core DTN protocol that + provides transmission reliability based on delay-tolerant + acknowledgments, timeouts, and retransmissions. The LTP + specification is defined in Internet RFC 5326. +- bp (Bundle Protocol), a core DTN protocol that provides + delay-tolerant forwarding of data through a network in which + continuous end-to-end connectivity is never assured, including + support for delay-tolerant dynamic routing. The BP specification is + defined in Internet RFC 5050. +- dgr (Datagram Retransmission), an alternative implementation of LTP + that is designed for use on the Internet. Equipped with algorithms + for TCP-like congestion control, DGR enables data to be transmitted + via UDP with reliability comparable to that provided by TCP. The dgr + system is provided primarily for the conveyance of Meta-AMS (see + below) protocol traffic in an Internet-like environment. +- ams (Asynchronous Message Service), an application-layer service + that is not part of the DTN architecture but utilizes underlying DTN + protocols. AMS comprises three protocols supporting the distribution + of brief messages within a network: + + - The core AAMS (Application AMS) protocol, which does message + distribution on both the publish/subscribe model and the + client/server model, as required by the application. + - The MAMS (Meta-AMS) protocol, which distributes control + information enabling the operation of the Application AMS + protocol. + - The RAMS (Remote AMS) protocol, which performs aggregated + message distribution to end nodes that may be numerous and/or + accessible only over very expensive links, using an aggregation + tree structure similar to the distribution trees used by + Internet multicast technologies. +- cfdp (CCSDS File Delivery Protocol), another application-layer + service that is not part of the DTN architecture but utilizes + underlying DTN protocols. CFDP performs the segmentation, + transmission, reception, reassembly, and delivery of files in a + delay-tolerant manner. ION's implementation of CFDP conforms to the + "class 1" definition of the protocol in the CFDP standard, utilizing + DTN (BP, nominally over LTP) as its "unitdata transport" layer. +- bss (Bundle Streaming Service), a system for efficient data + streaming over a delay-tolerant network. The bss package + includes (a) a convergence-layer protocol (bssp) that preserves + in-order arrival of all data that were never lost en route, yet + ensures that all data arrive at the destination eventually, and (b) + a library for building delay-tolerant streaming applications, which + enables low-latency presentation of streamed data received in real + time while offering rewind/playback capability for the entire stream + including late-arriving retransmitted data. +- tc (Trusted Collective), a system for propagating critical yet + non-confidential information in a trustworthy manner. tc can be + thought of as a delay-tolerant functional analog to the servers in + client/server architectures. Multiple applications may make use of + the tc system, but currently only one tc application is bundled with + ION: dtka (delay-tolerant key administration), which provides + delay-tolerant public key infrastructure. + +Taken together, the packages included in the ION software distribution +constitute a communication capability characterized by the following +operational features: + +- Reliable conveyance of data over a delay-tolerant network (*dtnet*), + i.e., a network in which it might never be possible for any node to + have reliable information about the detailed current state of any + other node. +- Built on this capability, reliable data streaming, reliable file + delivery, and reliable distribution of short messages to multiple + recipients (subscribers) residing in such a network. +- Management of traffic through such a network, taking into + consideration: + + - requirements for data security + - scheduled times and durations of communication opportunities + - fluctuating limits on data storage and transmission resources + - data rate asymmetry + - the sizes of application data units + - and user-specified final destination, priority, and useful + lifetime for those data units. +- Facilities for monitoring the performance of the network. +- Robustness against node failure. +- Portability across heterogeneous computing platforms. +- High speed with low overhead. +- Easy integration with heterogeneous underlying communication + infrastructure, ranging from Internet to dedicated spacecraft + communication links. + +### Constraints on the Design + +A DTN implementation intended to function in an interplanetary network +environment -- specifically, aboard interplanetary research spacecraft +separated from Earth and from one another by vast distances -- must +operate successfully within two general classes of design constraints: +link constraints and processor constraints. + +1. **Link constraints** + +All communications among interplanetary spacecraft are, obviously, +wireless. Less obviously, those wireless links are generally slow and +are usually asymmetric. + +The electrical power provided to on-board radios is limited and antennae +are relatively small, so signals are weak. This limits the speed at +which data can be transmitted intelligibly from an interplanetary +spacecraft to Earth, usually to some rate on the order of 256 Kbps to 6 +Mbps. + +The electrical power provided to transmitters on Earth is certainly much +greater, but the sensitivity of receivers on spacecraft is again +constrained by limited power and antenna mass allowances. Because +historically the volume of command traffic that had to be sent to +spacecraft was far less than the volume of telemetry the spacecraft were +expected to return, spacecraft receivers have historically been +engineered for even lower data rates from Earth to the spacecraft, on +the order of 1 to 2 Kbps. + +As a result, the cost per octet of data transmission or reception is +high and the links are heavily subscribed. Economical use of +transmission and reception opportunities is therefore important, and +transmission is designed to enable useful information to be obtained +from brief communication opportunities: units of transmission are +typically small, and the immediate delivery of even a small part +(carefully delimited) of a large data object may be preferable to +deferring delivery of the entire object until all parts have been +acquired. + +2. **Processor constraints** + +The computing capability aboard a robotic interplanetary spacecraft is +typically quite different from that provided by an engineering +workstation on Earth. In part this is due, again, to the limited +available electrical power and limited mass allowance within which a +flight computer must operate. But these factors are exacerbated by the +often intense radiation environment of deep space. In order to minimize +errors in computation and storage, flight processors must be +radiation-hardened and both dynamic memory and non-volatile storage +(typically flash memory) must be radiation-tolerant. The additional +engineering required for these adaptations takes time and is not +inexpensive, and the market for radiation-hardened spacecraft computers +is relatively small; for these reasons, the latest advances in +processing technology are typically not available for use on +interplanetary spacecraft, so flight computers are invariably slower +than their Earth-bound counterparts. As a result, the cost per +processing cycle is high and processors are heavily subscribed; +economical use of processing resources is very important. + +The nature of interplanetary spacecraft operations imposes a further +constraint. These spacecraft are wholly robotic and are far beyond the +reach of mission technicians; hands-on repairs are out of the question. +Therefore the processing performed by the flight computer must be highly +reliable, which in turn generally means that it must be highly +predictable. Flight software is typically required to meet "hard" +real-time processing deadlines, for which purpose it must be run within +a hard real-time operating system (RTOS). + +One other implication of the requirement for high reliability in flight +software is that the dynamic allocation of system memory may be +prohibited except in certain well-understood states, such as at system +start-up. Unrestrained dynamic allocation of system memory introduces a +degree of unpredictability into the overall flight system that can +threaten the reliability of the computing environment and jeopardize the +health of the vehicle. + +### Design Principles + +The design of the ION software distribution reflects several core +principles that are intended to address these constraints. + +![Figure 2](./media-ion-guide/media/image14.png) + +**Figure 2 ION inter-task communication** + +1. **Shared memory** + +Since ION must run on flight processors, it had to be designed to +function successfully within an RTOS. Many real-time operating systems +improve processing determinism by omitting the support for +protected-memory models that is provided by Unix-like operating systems: +all tasks have direct access to all regions of system memory. (In +effect, all tasks operate in kernel mode rather than in user mode.) ION +therefore had to be designed with no expectation of memory protection. + +But universally shared access to all memory can be viewed not only as a +hazard but also as an opportunity. Placing a data object in shared +memory is an extremely efficient means of passing data from one software +task to another. + +ION is designed to exploit this opportunity as fully as possible. In +particular, virtually all inter-task data interchange in ION follows the +model shown in Figure 2: + +- The sending task takes a mutual exclusion semaphore (mutex) + protecting a linked list in shared memory (either DRAM or + non-volatile memory), appends a data item to the list, releases the + mutex, and gives a "signal" semaphore associated with the list to + announce that the list is now non-empty. +- The receiving task, which is already pending on the linked list's + associated signal semaphore, resumes execution when the semaphore is + given. It takes the associated mutex, extracts the next data item + from the list, releases the mutex, and proceeds to operate on the + data item from the sending task. + +Semaphore operations are typically extremely fast, as is the storage and +retrieval of data in memory, so this inter-task data interchange model +is suitably efficient for flight software. + +1. **Zero-copy procedures** + +Given ION's orientation toward the shared memory model, a further +strategy for processing efficiency offers itself: if the data item +appended to a linked list is merely a pointer to a large data object, +rather than a copy, then we can further reduce processing overhead by +eliminating the cost of byte-for-byte copying of large objects. + +Moreover, in the event that multiple software elements need to access +the same large object at the same time, we can provide each such +software element with a pointer to the object rather than its own copy +(maintaining a count of references to assure that the object is not +destroyed until all elements have relinquished their pointers). This +serves to reduce somewhat the amount of memory needed for ION +operations. + +2. **Highly distributed processing** + +The efficiency of inter-task communications based on shared memory makes +it practical to distribute ION processing among multiple relatively +simple pipelined tasks rather than localize it in a single, somewhat +more complex daemon. This strategy has a number of advantages: + +- The simplicity of each task reduces the sizes of the software + modules, making them easier to understand and maintain, and thus it + can somewhat reduce the incidence of errors. +- The scope of the ION operating stack can be adjusted incrementally + at run time, by spawning or terminating instances of configurable + software elements, without increasing the size or complexity of any + single task and without requiring that the stack as a whole be + halted and restarted in a new configuration. In theory, a module + could even be upgraded with new functionality and integrated into + the stack without interrupting operations. +- The clear interfaces between tasks simplify the implementation of + flow control measures to prevent uncontrolled resource consumption. + + 1. **Portability** + +Designs based on these kinds of principles are foreign to many software +developers, who may be far more comfortable in development environments +supported by protected memory. It is typically much easier, for example, +to develop software in a Linux environment than in VxWorks 5.4. However, +the Linux environment is not the only one in which ION software must +ultimately run. + +Consequently, ION has been designed for easy portability. POSIX™ API +functions are widely used, and differences in operating system support +that are not concealed within the POSIX abstractions are mostly +encapsulated in two small modules of platform-sensitive ION code. The +bulk of the ION software runs, without any source code modification +whatsoever, equally well in Linux™ (Red Hat®, Fedora™, and Ubuntu™, so +far), FreeBSD®, Solaris® 9, Microsoft Windows (the MinGW environment), +OS/X®, VxWorks® 5.4, and RTEMS™, on both 32-bit and 64-bit processors. +Developers may compile and test ION modules in whatever environment they +find most convenient. + +### Organizational Overview + +Two broad overviews of the organization of ION may be helpful at this +point. First, here is a summary view of the main functional dependencies +among ION software elements: + +![Figure 3](./media-ion-guide/media/image8.png) + +**Figure 3 ION software functional dependencies** + +That is, BP and LTP invoke functions provided by the sdr, zco, psm, and +platform elements of the ici package, in addition to functions provided +by the operating system itself; the zco functions themselves also invoke +sdr, psm, and platform functions; and so on. + +Second, here is a summary view of the main line of data flow in ION's +DTN protocol implementations: + +![Figure 4](./media-ion-guide/media/image1.png) + +**Figure 4 Main line of ION data flow** + +Note that data objects residing in shared memory, many of them in a +nominally non-volatile SDR data store, constitute the central organizing +principle of the design. Here as in other diagrams showing data flow in +this document: + +- Ordered collections of data objects are shown as cylinders. +- Darker greyscale data entities indicate data that are managed in the + SDR data store, while lighter greyscale data entities indicate data + that are managed in volatile DRAM to improve performance. +- Rectangles indicate processing elements (tasks, processes, threads), + sometimes with library references specifically identified. + +A few notes on this main line data flow: + +- For simplicity, the data flow depicted here is a "loopback" flow in + which a single BP "node" is shown sending data to itself (a useful + configuration for test purposes). To depict typical operations over + a network we would need two instances of this node diagram, such + that the \ task of one node is shown sending data to the + \ task of the other and vice versa. +- A BP application or application service (such as Remote AMS) that + has access to the local BP node -- for our purposes, the "sender" -- + invokes the bp_send function to send a unit of application data to a + remote counterpart. The destination of the application data unit is + expressed as a BP endpoint ID (EID). The application data unit is + encapsulated in a bundle and is queued for forwarding. +- The forwarder task identified by the "scheme" portion of the + bundle's destination EID removes the bundle from the forwarding + queue and computes a route to the destination EID. The first node on + the route is termed the "proximate node" for the computed route. The + forwarder appends the bundle to the transmission queue for the + convergence-layer manager (CLM) daemon that is responsible for + transmission to the proximate node. +- The CLM daemon removes the bundle from the transmission queue and + imposes rate control, fragments the bundle as necessary, and appends + the bundle to the transmission buffer for some underlying + "convergence layer" (CL) protocol interface to the proximate node, + termed an *outduct*. In the event that multiple outducts are + available for transmission to that node (e.g., multiple radio + frequency bands), the CLM invokes mission-supplied code to select + the appropriate duct. Each outduct is serviced by some CL-specific + output task that communicates with the proximate node -- in this + case, the LTP output task **ltpclo**. (Other CL protocols supported + by ION include TCP and UDP.) +- The output task for LTP transmission to the selected proximate node + removes the bundle from the transmission buffer and invokes the + ltp_send function to append it to a *block* that is being assembled + for transmission to the proximate node. (Because LTP acknowledgement + traffic is issued on a per-block basis, we can limit the amount of + acknowledgement traffic on the network by aggregating multiple + bundles into a single block rather than transmitting each bundle in + its own block.) +- The **ltpmeter** task for the selected proximate node divides the + aggregated block into multiple segments and enqueues them for + transmission by underlying link-layer transmission software, such as + an implementation of the CCSDS AOS protocol. +- Underlying link-layer software at the sending node transmits the + segments to its counterpart at the proximate node (the receiver), + where they are used to reassemble the transmission block. +- The receiving node's input task for LTP reception extracts the + bundles from the reassembled block and dispatches them: each bundle + whose final destination is some other node is queued for forwarding, + just like bundles created by local applications, while each bundle + whose final destination is the local node is queued for delivery to + whatever application "opens" the BP endpoint identified by the + bundle's final destination endpoint ID. (Note that a multicast + bundle may be both queued for forwarding, possibly to multiple + neighboring nodes, and also queued for delivery.) +- The destination application or application service at the receiving + node opens the appropriate BP endpoint and invokes the bp_receive + function to remove the bundle from the associated delivery queue and + extract the original application data unit, which it can then + process. + +Finally, note that the data flow shown here represents the sustained +operational configuration of a node that has been successfully +instantiated on a suitable computer. The sequence of operations +performed to reach this configuration is not shown. That startup +sequence will necessarily vary depending on the nature of the computing +platform and the supporting link services. Broadly, the first step +normally is to run the **ionadmin** utility program to initialize the +data management infrastructure required by all elements of ION. +Following this initialization, the next steps normally are (a) any +necessary initialization of link service protocols, (b) any necessary +initialization of convergence-layer protocols (e.g., LTP -- the +**ltpadmin** utility program), and finally (c) initialization of the +Bundle Protocol by means of the **bpadmin** utility program. BP +applications should not try to commence operation until BP has been +initialized. + +### Resource Management in ION + +Successful Delay-Tolerant Networking relies on retention of bundle +protocol agent state information -- including protocol traffic that is +awaiting a transmission opportunity -- for potentially lengthy +intervals. The nature of that state information will fluctuate rapidly +as the protocol agent passes through different phases of operation, so +efficient management of the storage resources allocated to state +information is a key consideration in the design of ION. + +Two general classes of storage resources are managed by ION: volatile +"working memory" and non-volatile "heap". + +1. **Working Memory** + +ION's "working memory" is a fixed-size pool of shared memory (dynamic +RAM) that is allocated from system RAM at the time the bundle protocol +agent commences operation. Working memory is used by ION tasks to store +temporary data of all kinds: linked lists, red-black trees, transient +buffers, volatile databases, etc. All intermediate data products and +temporary data structures that ought not to be retained in the event of +a system power cycle are written to working memory. + +Data structures residing in working memory may be shared among ION tasks +or may be created and managed privately by individual ION tasks. The +dynamic allocation of working memory to ION tasks is accomplished by the +Personal Space Management (PSM) service, described later. All of the +working memory for any single ION bundle protocol agent is managed as a +single PSM "partition". The size of the partition is specified in the +**wmSize** parameter of the ionconfig file supplied at the time ION is +initialized. + +2. **Heap** + +ION's "heap" is a fixed-size pool of notionally non-volatile storage +that is likewise allocated at the time the bundle protocol agent +commences operation. This notionally non-volatile space **may** occupy a +fixed-size pool of shared memory (dynamic RAM, which might or might not +be battery-backed), or it **may** occupy only a single fixed-size file +in the file system, or it may occupy both. In the latter case, all heap +data are written both to memory and to the file but are read only from +memory; this configuration offers the reliable non-volatility of file +storage coupled with the high performance of retrieval from dynamic RAM. + +We characterize ION's heap storage as "notionally" non-volatile because +the heap may be configured to reside only in memory (or, for that +matter, in a file that resides in the file system of a RAM disk). When +the heap resides only in memory, its contents are truly non-volatile +only if that memory is battery-backed. Otherwise heap storage is in +reality as volatile as working memory: heap contents will be lost upon a +system power cycle (which may in fact be the preferred behavior for any +given deployment of ION). However, the heap should not be thought of as +\"memory\" even when it in fact resides only in DRAM, just as a disk +device should not be thought of as \"memory\" even when it is in fact a +RAM disk. + +![](./media-ion-guide/media/image18.png){width="4.738575021872266in" +height="3.338542213473316in"} + +**Figure 5 ION heap space use** + +The ION heap is used for storage of data that (in at least some +deployments) would have to be retained in the event of a system power +cycle to ensure the correct continued operation of the node. For +example, all queues of bundles awaiting route computation, transmission, +or delivery reside in the node's heap. So do the non-volatile databases +for all of the protocols implemented within ION, together with all of +the node's persistent configuration parameters. + +The dynamic allocation of heap space to ION tasks is accomplished by the +Simple Data Recorder (SDR) service, described later. The entire heap for +any single ION bundle protocol agent is managed as a single SDR "data +store". + +Space within the ION heap is apportioned as shown in Figure 5. The total +number of bytes of storage space in the heap is computed as the product +of the size of a "word" on the deployment platform (normally the size of +a pointer) multiplied by the value of the **heapWords** parameter of the +ionconfig file supplied at the time ION is initialized. Of this total, +20% is normally reserved as margin and another 40% is normally reserved +for various infrastructure operations. (Both of these percentages are +macros that may be overridden at compile time.) The remainder is +available for storage of protocol state data in the form of "zero-copy +objects", described later. At any given moment, the data encapsulated in +a zero-copy object may "belong" to any one of the protocols in the ION +stack (AMS, CFDP, BP, LTP), depending on processing state; the available +heap space is a single common resource to which all of the protocols +share concurrent access. + +Because the heap is used to store queues of bundles awaiting processing, +blocks of LTP data awaiting transmission or reassembly, etc., the heap +for any single ION node must be large enough to contain the maximum +volume of such data that the node will be required to retain during +operations. Demand for heap space is substantially mitigated if most of +the application data units passed to ION for transmission are +file-resident, as the file contents themselves need not be copied into +the heap. In general, however, computing the optimum ION heap size for a +given deployment remains a research topic. + +### Package Overviews + +#### **Interplanetary Communication Infrastructure (ICI)** + +The ICI package in ION provides a number of core services that, from +ION's point of view, implement what amounts to an extended POSIX-based +operating system. ICI services include the following: + +1\. Platform + +The platform system contains operating-system-sensitive code that +enables ICI to present a single, consistent programming interface to +those common operating system services that multiple ION modules +utilize. For example, the platform system implements a standard +semaphore abstraction that may invisibly be mapped to underlying POSIX +semaphores, SVR4 IPC semaphores, Windows Events, or VxWorks semaphores, +depending on which operating system the package is compiled for. The +platform system also implements a standard shared-memory abstraction, +enabling software running on operating systems both with and without +memory protection to participate readily in ION's shared-memory-based +computing environment. + +2\. Personal Space Management (PSM) + +Although sound flight software design may prohibit the uncontrolled +dynamic management of system memory, private management of assigned, +fixed blocks of system memory is standard practice. Often that private +management amounts to merely controlling the reuse of fixed-size rows in +static tables, but such techniques can be awkward and may not make the +most efficient use of available memory. The ICI package provides an +alternative, called PSM, which performs high-speed dynamic allocation +and recovery of variable-size memory objects within an assigned memory +block of fixed size. A given PSM-managed memory block may be either +private or shared memory. + +3\. Memmgr + +The static allocation of privately-managed blocks of system memory for +different purposes implies the need for multiple memory management +regimes, and in some cases a program that interacts with multiple +software elements may need to participate in the private shared-memory +management regimes of each. ICI's memmgr system enables multiple memory +managers -- for multiple privately-managed blocks of system memory -- to +coexist within ION and be concurrently available to ION software +elements. + +4\. Lyst + +The lyst system is a comprehensive, powerful, and efficient system for +managing doubly-linked lists in private memory. It is the model for a +number of other list management systems supported by ICI; as noted +earlier, linked lists are heavily used in ION inter-task communication. + +5\. Llcv + +The llcv (Linked-List Condition Variables) system is an inter-thread +communication abstraction that integrates POSIX thread condition +variables (vice semaphores) with doubly-linked lists in private memory. + +6\. Smlist + +Smlist is another doubly-linked list management service. It differs from +lyst in that the lists it manages reside in shared (rather than private) +DRAM, so operations on them must be semaphore-protected to prevent race +conditions. + +7\. SmRbt + +The SmRbt service provides mechanisms for populating and navigating +"red/black trees" (RBTs) residing in shared DRAM. RBTs offer an +alternative to linked lists: like linked lists they can be navigated as +queues, but locating a single element of an RBT by its "key" value can +be much quicker than the equivalent search through an ordered linked +list. + +8\. Simple Data Recorder (SDR) + +SDR is a system for managing non-volatile storage, built on exactly the +same model as PSM. Put another way, SDR is a small and simple +"persistent object" system or "object database" management system. It +enables straightforward management of linked lists (and other data +structures of arbitrary complexity) in non-volatile storage, notionally +within a single file whose size is pre-defined and fixed. + +SDR includes a transaction mechanism that protects database integrity by +ensuring that the failure of any database operation will cause all other +operations undertaken within the same transaction to be backed out. The +intent of the system is to assure retention of coherent protocol engine +state even in the event of an unplanned flight computer reboot in the +midst of communication activity. + +9\. Sptrace + +The sptrace system is an embedded diagnostic facility that monitors the +performance of the PSM and SDR space management systems. It can be used, +for example, to detect memory "leaks" and other memory management +errors. + +10\. Zco + +ION's zco (zero-copy objects) system leverages the SDR system's storage +flexibility to enable user application data to be encapsulated in any +number of layers of protocol without copying the successively augmented +protocol data unit from one layer to the next. It also implements a +reference counting system that enables protocol data to be processed +safely by multiple software elements concurrently -- e.g., a bundle may +be both delivered to a local endpoint and, at the same time, queued for +forwarding to another node -- without requiring that distinct copies of +the data be provided to each element. + +11\. Rfx + +The ION rfx (R/F Contacts) system manages lists of scheduled +communication opportunities in support of a number of LTP and BP +functions. + +12\. Ionsec + +The IONSEC (ION security) system manages information that supports the +implementation of security mechanisms in the other packages: security +policy rules and computation keys. + +#### **Licklider Transmission Protocol (LTP)** + +The ION implementation of LTP conforms fully to RFC 5326, but it also +provides two additional features that enhance functionality without +affecting interoperability with other implementations: + +- The service data units -- nominally bundles -- passed to LTP for + transmission may be aggregated into larger blocks before + segmentation. By controlling block size we can control the volume of + acknowledgement traffic generated as blocks are received, for + improved accommodation of highly asynchronous data rates. +- The maximum number of transmission sessions that may be concurrently + managed by LTP (a protocol control parameter) constitutes a + transmission "window" -- the basis for a delay-tolerant, + non-conversational flow control service over interplanetary links. + +In the ION stack, LTP serves effectively the same role that is performed +by an LLC protocol (such as IEEE 802.2) in the Internet architecture, +providing flow control and retransmission-based reliability between +topologically adjacent bundle protocol agents. + +All LTP session state is safely retained in the ION heap for rapid +recovery from a spacecraft or software fault. + +#### **Bundle Protocol (BP)** + +The ION implementation of BP conforms fully to RFC 5050, including +support for the following standard capabilities: + +- Prioritization of data flows +- Proactive bundle fragmentation +- Bundle reassembly from fragments +- Flexible status reporting +- Custody transfer, including re-forwarding of custodial bundles upon + timeout interval expiration or failure of nominally reliable + convergence-layer transmission + +The system also provides three additional features that enhance +functionality without affecting interoperability with other +implementations: + +- Rate control provides support for congestion forecasting and + avoidance. +- Bundle headers are encoded into compressed form (CBHE, as noted + earlier) before issuance, to reduce protocol overhead and improve + link utilization. +- Bundles may be "multicast" to all nodes that have registered within + a given multicast group endpoint. + +In addition, ION BP includes a system for computing dynamic routes +through time-varying network topology assembled from scheduled, bounded +communication opportunities. This system, called "Contact Graph +Routing," is described later in this Guide. + +In short, BP serves effectively the same role that is performed by IP in +the Internet architecture, providing route computation, forwarding, +congestion avoidance, and control over quality of service. + +All bundle transmission state is safely retained in the ION heap for +rapid recovery from a spacecraft or software fault. + +#### **Asynchronous Message Service (AMS)** + +The ION implementation of the CCSDS AMS standard conforms fully to CCSDS +735.0-B-1. AMS is a data system communications architecture under which +the modules of mission systems may be designed as if they were to +operate in isolation, each one producing and consuming mission +information without explicit awareness of which other modules are +currently operating. Communication relationships among such modules are +self-configuring; this tends to minimize complexity in the development +and operations of modular data systems. + +A system built on this model is a "society" of generally autonomous +inter-operating modules that may fluctuate freely over time in response +to changing mission objectives, modules' functional upgrades, and +recovery from individual module failure. The purpose of AMS, then, is to +reduce mission cost and risk by providing standard, reusable +infrastructure for the exchange of information among data system modules +in a manner that is simple to use, highly automated, flexible, robust, +scalable, and efficient. + +A detailed discussion of AMS is beyond the scope of this Design Guide. +For more information, please see the [AMS Programmer's Guide.](AMS-Programmer-Guide.md) + +#### **Datagram Retransmission (DGR)** + +The DGR package in ION is an alternative implementation of LTP that is +designed to operate responsibly -- i.e., with built-in congestion +control -- in the Internet or other IP-based networks running UDP. It is provided as +a candidate "primary transfer service" in support of AMS operations in +an Internet-like (non-delay-tolerant) environment. The DGR design +combines LTP's concept of concurrent transmission transactions with +congestion control and timeout interval computation algorithms adapted +from TCP. DGR includes an adaptive timeout-interval-computing component that provides +maximum opportunity for reception of acknowledgements, minimizing retransmission. +By monitoring changes in the rate at which message-transmission transactions +are completed, DGR detects changes in the level of congestion and +responds by imposing varying degrees of delay on the transmission of +new messages. In addition, DGR maximizes throughput by not waiting +for acknowledgement of a message before sending the next message. All +DGR communication is asynchronous, to maximize efficient utilization of +network connections. DGR manages multiple concurrent datagram transmission +and acknowledgement conversations. + +#### **CCSDS File Delivery Protocol (CFDP)** + +The ION implementation of CFDP conforms fully to Service Class 1 +(Unreliable Transfer) of CCSDS 727.0-B-4, including support for the +following standard capabilities: + +- Segmentation of files on user-specified record boundaries. +- Transmission of file segments in protocol data units that are + conveyed by an underlying Unitdata Transfer service, in this case + the DTN protocol stack. File data segments may optionally be + protected by CRCs. When the DTN protocol stack is configured for + reliable data delivery (i.e., with BP custody transfer running over + a reliable convergence-layer protocol such as LTP), file delivery is + reliable; CFDP need not perform retransmission of lost data itself. +- Reassembly of files from received segments, possibly arriving over a + variety of routes through the delay-tolerant network. The integrity + of the delivered files is protected by checksums. +- User-specified fault handling procedures. +- Operations (e.g., directory creation, file renaming) on remote file + systems. + +All CFDP transaction state is safely retained in the ION heap for rapid +recovery from a spacecraft or software fault. + +#### **Bundle Streaming Service (BSS)** + +The BSS service provided in ION enables a stream of video, audio, or +other continuously generated application data units, transmitted over a +delay-tolerant network, to be presented to a destination application in +two useful modes concurrently: + +- In the order in which the data units were generated, with the least + possible end-to-end delivery latency, but possibly with some gaps + due to transient data loss or corruption. +- In the order in which the data units were generated, without gaps + (i.e., including lost or corrupt data units which were omitted from + the real-time presentation but were subsequently retransmitted), but + in a non-real-time "playback" mode. + +#### **Trusted Collective (TC)** + +The TC service provided in ION enables critical but non-confidential +information (such as public keys, for asymmetric cryptography) to be +provided in a delay-tolerant, trustworthy manner. An instance of TC +comprises: + +- A distributed Authority, the members of which must reach consensus + on database content and must collaborate on the proactive + distribution of that content. +- Any number of Clients, which: + + - Announce new content to the Authority via authenticated bundle + multicast, and/or + - Receive trustworthy bulletins multicast by the members of the + Authority. + +#### Acronyms + +| Acronyms | Description | +| -------- | --------------------------------------------- | +| BP | Bundle Protocol | +| BSP | Bundle Security Protocol | +| BSS | Bundle Streaming Service | +| CCSDS | Consultative Committee for Space Data Systems | +| CFDP | CCSDS File Delivery Protocol | +| CGR | Contact Graph Routing | +| CL | convergence layer | +| CLI | convergence layer input | +| CLO | convergence layer output | +| DTKA | Delay-Tolerant Key Administration | +| DTN | Delay-Tolerant Networking | +| ICI | Interplanetary Communication Infrastructure | +| ION | Interplanetary Overlay Network | +| LSI | link service input | +| LSO | link service output | +| LTP | Licklider Transmission Protocol | +| OWLT | one-way light time | +| RFC | request for comments | +| RFX | Radio (R/F) Contacts | +| RTT | round-trip time | +| TC | Trusted Collective | +| TTL | time to live | + +### Network Operation Concepts + +A small number of network operation design elements -- fragmentation and +reassembly, bandwidth management, and delivery assurance +(retransmission) -- can potentially be addressed at multiple layers of +the protocol stack, possibly in different ways for different reasons. In +stack design it's important to allocate this functionality carefully so +that the effects at lower layers complement, rather than subvert, the +effects imposed at higher layers of the stack. This allocation of +functionality is discussed below, together with a discussion of several +related key concepts in the ION design. + +#### **Fragmentation and Reassembly** + +To minimize transmission overhead and accommodate asymmetric links +(i.e., limited "uplink" data rate from a ground data system to a +spacecraft) in an interplanetary network, we ideally want to send +"downlink" data in the largest possible aggregations -- coarse-grained +transmission. + +But to minimize head-of-line blocking (i.e., delay in transmission of a +newly presented high-priority item) and minimize data delivery latency +by using parallel paths (i.e., to provide fine-grained partial data +delivery, and to minimize the impact of unexpected link termination), we +want to send "downlink" data in the smallest possible aggregations -- +fine-grained transmission. + +We reconcile these impulses by doing both, but at different layers of +the ION protocol stack. + +First, at the application service layer (AMS and CFDP) we present +relatively small application data units (ADUs) -- on the order of 64 KB +-- to BP for encapsulation in bundles. This establishes an upper bound +on head-of-line blocking when bundles are de-queued for transmission, +and it provides perforations in the data stream at which forwarding can +readily be switched from one link (route) to another, enabling partial +data delivery at relatively fine, application-appropriate granularity. + +(Alternatively, large application data units may be presented to BP and +the resulting large bundles may be proactively fragmented at the time +they are presented to the convergence-layer manager. This capability is +meant to accommodate environments in which the convergence-layer manager +has better information than the application as to the optimal bundle +size, such as when the residual capacity of a contact is known to be +less than the size of the bundle.) + +Then, at the BP/LTP convergence layer adapter lower in the stack, we +aggregate these small bundles into *blocks* for presentation to LTP: + +Any continuous sequence of bundles that are to be shipped to the same +LTP engine and all require assured delivery may be aggregated into a +single block, to reduce overhead and minimize report traffic. + +However, this aggregation is constrained by an aggregation size limit +rule: aggregation must stop and the block must be transmitted as soon as +the sum of the sizes of all bundles aggregated into the block exceeds +the block aggregation threshhold value declared for the applicable +*span* (the relationship between the local node's LTP engine and the +receiving LTP engine) during LTP protocol configuration via +**ltpadmin**. + +Given a preferred block acknowledgment period -- e.g., a preferred +acknowledgement traffic rate of one report per second -- the nominal +block aggregation threshold is notionally computed as the amount of data +that can be sent over the link to the receiving LTP engine in a single +block acknowledgment period at the planned outbound data rate to that +engine. + +Taken together, application-level fragmentation (or BP proactive +fragmentation) and LTP aggregation place an upper limit on the amount of +data that would need to be re-transmitted over a given link at next +contact in the event of an unexpected link termination that caused +delivery of an entire block to fail. For example, if the data rate is 1 +Mbps and the nominal block size is 128 KB (equivalent to 1 second of +transmission time), we would prefer to avoid the risk of having wasted +five minutes of downlink in sending a 37.5 MB file that fails on +transmission of the last kilobyte, forcing retransmission of the entire +37.5 MB. We therefore divide the file into, say, 1200 bundles of 32 KB +each which are aggregated into blocks of 128 KB each: only a single +block failed, so only that block (containing just 4 bundles) needs to be +retransmitted. The cost of this retransmission is only 1 second of link +time rather than 5 minutes. By controlling the cost of convergence-layer +protocol failure in this way, we avoid the overhead and complexity of +"reactive fragmentation" in the BP implementation. + +Finally, within LTP itself we fragment the block as necessary to +accommodate the Maximum Transfer Unit (MTU) size of the underlying link +service, typically the transfer frame size of the applicable CCSDS link +protocol. + +#### **Bandwidth Management** + +The allocation of bandwidth (transmission opportunity) to application +data is requested by the application task that's passing data to DTN, +but it is necessarily accomplished only at the lowest layer of the stack +at which bandwidth allocation decisions can be made -- and then always +in the context of node policy decisions that have global effect. + +The transmission queue interface to a given neighbor in the network is +actually three queues of outbound bundles rather than one: one queue for +each of the defined levels of priority ("class of service") supported by +BP. When an application presents an ADU to BP for encapsulation in a +bundle, it indicates its own assessment of the ADU's priority. Upon +selection of a proximate forwarding destination node for that bundle, +the bundle is appended to whichever of the queues corresponds to the +ADU's priority. + +Normally the convergence-layer manager (CLM) task servicing a given +proximate node extracts bundles in strict priority order from the heads +of the three queues. That is, the bundle at the head of the +highest-priority non-empty queue is always extracted. + +However, if the ION_BANDWIDTH_RESERVED compiler option is selected at +the time ION is built, the convergence-layer manager task servicing a +given proximate node extracts bundles in interleaved fashion from the +heads of the node's three queues: + +- Whenever the priority-2 ("express") queue is non-empty, the bundle + at the head of that queue is the next one extracted. +- At all other times, bundles from both the priority-1 queue and the + priority-0 queue are extracted, but over a given period of time + twice as many bytes of priority-1 bundles will be extracted as bytes + of priority-0 bundles. + +Following insertion of the extracted bundles into transmission buffers, +CLO tasks other than **ltpclo** simply segment the buffered bundles as +necessary and transmit them using the underlying convergence-layer +protocols. In the case of **ltpclo**, the output task aggregates the +buffered bundles into blocks as described earlier and a second daemon +task named **ltpmeter** waits for aggregated blocks to be completed; +**ltpmeter**, rather than the CLO task itself, segments each completed +block as necessary and passes the segments to the link service protocol +that underlies LTP. Either way, the transmission ordering requested by +application tasks is preserved. + +#### **Contact Plans** + +In the Internet, protocol operations can be largely driven by currently +effective information that is discovered opportunistically and +immediately, at the time it is needed, because the latency in +communicating this information over the network is negligible: distances +between communicating entities are small and connectivity is continuous. +In a DTN-based network, however, ad-hoc information discovery would in +many cases take so much time that it could not be completed before the +information lost currency and effectiveness. Instead, protocol +operations must be largely driven by information that is pre-placed at +the network nodes and tagged with the dates and times at which it +becomes effective. This information takes the form of *contact plans* +that are managed by the R/F Contacts (rfx) services of ION's ici +package. + +![Figure 6](./media-ion-guide/media/image6.png) + +**Figure 6 RFX services in ION** + +The structure of ION's RFX (contact plan) database, the rfx system +elements that populate and use that data, and affected portions of the +BP and LTP protocol state databases are shown in Figure 6. (For +additional details of BP and LTP database management, see the BP/LTP +discussion later in this document.) + +To clarify the notation of this diagram, which is also used in other +database structure diagrams in this document: + +- Data objects of defined structure are shown as circles. Dark + greyscale indicates notionally non-volatile data retained in "heap" + storage, while lighter greyscale indicates volatile data retained in + dynamic random access memory. +- Solid arrows connecting circles indicate one-to-many cardinality. +- A dashed arrow between circles indicates a potentially many-to-one + reference mapping. +- Arrows from processing elements (rectangles) to data entities + indicate data production, while arrows from data entities to + processing elements indicate data retrieval. + +A *contact* is here defined as an interval during which it is expected +that data will be transmitted by DTN node A (the contact's transmitting +node) and most or all of the transmitted data will be received by node B +(the contact's receiving node). Implicitly, the transmitting mode will +utilize some "convergence-layer" protocol underneath the Bundle Protocol +to effect this direct transmission of data to the receiving node. Each +contact is characterized by its start time, its end time, the identities +of the transmitting and receiving nodes, and the rate at which data are +expected to be transmitted by the transmitting node throughout the +indicated time period. + +(Note that a contact is specifically ***not*** an episode of activity on +a link. Episodes of activity on different links -- e.g., different radio +transponders operating on the same spacecraft -- may well overlap, but +contacts by definition cannot; they are bounded time intervals and as +such are innately "tiled". For example, suppose transmission on link X +from node A to node B, at data rate RX, begins at time T1 and ends at +time T2; also, transmission on link Y from node A to node B, at data +rate RY begins at time T3 and ends at time T4. If T1 = T3 and T2 = T4, +then there is a single contact from time T1 to time T2 at data rate RX + +RY. If T1 \< T3 and T2 = T4, then there are two contiguous contacts: one +from T1 to T3 at data rate RX, then one from T3 to T2 at data rate RX + +RY. If T1 \< T3 and T3\