results.tex~

\chapter{Experimental Evaluation}
\label{chap:results}

Our proposed system requires wireless devices with multiple networking interface modules. Each of these modules must also have cognitive capability to ensure the basic requirements of our proposed architecture. The development of such devices involves a highly complex level of sophistication and fabrication. Such a development of cognitive radio networks in real setup is still under research. Therefore, we evaluate the performance of our proposed feedback-based multi-radio exploitation approach through extensive discrete-event simulation using \texttt{ns-3}. Yet, we have to make several modifications on the \texttt{ns-3} simulator to evaluate our proposed approach on MRCRNs.

\section{Simulator Modifications}

We implement our proposed approach on top of the Cognitive radio extension for ns-3 namely CRE-NS3~\cite{al2014simulating}. We modify the cognitive module of CRE-NS3 to incorporate our feedback-based approach. The existing cognitive module of CRE-NS3 provides three interfaces for each device namely control interface, transmitter interface, and receiver interface. The transmitter and receiver interfaces of the module emulate a real cognitive transceivers. Therefore, we introduce the functionality of varying number of cognitive transceivers through varying the number of the transmitter and receiver interfaces.

To implement this functionality, we utilize the \texttt{Callback} mechanism of \texttt{ns-3} extensively. Using this mechanism, we make sure that our counters (\texttt{pktQueued}, \texttt{pktSent}, \texttt{pktTransmitted}, and \texttt{pktReceived}) are incremented after corresponding events. The \texttt{Callback} mechanism has also been used to update \texttt{radioStatus}, \texttt{availableChannels}, and \texttt{currentChannel} lists.

 We also employ \texttt{ns-3} flow tagging feature, \texttt{FlowIdTag} to encapsulate and extract extra information to and from packets. As multiple radios on a single SU node share the same upper layer address (IP address), the extra \texttt{FlowIdTag} of each packet determines the radio reference (sender and receiver), using which upper layers can distinguish among multiple radios. Moreover, we add \texttt{DelayJitterEstimationTimestampTag} to each packet to calculate delay each packet experiences.

Apart from these changes, we have also made several changes in the \texttt{wifi} module of the \texttt{ns-3} simulator. Specifically, we have modified the \texttt{YansWifiPhy}, \texttt{YansWifiChannel}, \texttt{WifiPhyStateHelper}, \texttt{RegularWifiMac}, and \texttt{WifiNetDevice} models of the \texttt{wifi} module to add the cross-layer implementation of the multi-radio functionailty.

Using the modified simulator, we implement our proposed approach and evaluate its performance on the basis of four performance metrics -- total network throughput, end-to-end delay, packet drop ratio, and application layer packet delivery ratio. Besides, we measure values of these metrics for two existing MRCRN protocols and compared them against that obtained using several variants of our proposed approach. We briefly describe our simulation settings next before presenting the evaluation results.

\section{Simulation Settings}

We consider that arrival and departure of a PU follow a Poisson process~\cite{heo2008mathematical}. Accordingly, we consider an exponential distribution for both inter-arrival time and service time. Hence we adopt the mean time between two successive arrivals to be 5 seconds and the mean service time to be 2 seconds. Besides, we consider that each secondary user enables a constant bit rate application where the data transmission rate is varied from 1 Mbps to 32 Mbps. Here, each secondary user is equipped with a variable number of radios. Each of the radios consists of one transmitter interface and one receiver interface. The transmitter interface transmits data over any of the eleven orthogonal channels that conventionally operate with OFDM WiFi mode having 18Mbps data rate. For each transmitter interface or radio, we associate a drop-tail queue with a maximum capacity of 100 packets, each of 1KB in size. These interfaces have a transmission range of 130m and a sensing range of 250m. To ensure that the destination users are reachable from the source users, we place the destinations at an average distance of 80m from the sources. Maintaining such average distance, primary users and secondary users are placed randomly in an area of $500$m$\times 500$m. Here, we vary the number of secondary users from 12 to 40 with a granularity of 4. For each such settings, we perform 99 simulation iterations and then take average results of all the iterations. It is to be noted here that the maximum iteration count for obtaining 95\% confidence interval according to Monte Carlo Sampling~\cite{winston2000simulation} is found to be 61 in our experiment settings. As our initial simulation results shows that almost all the performance metrics become constant after 40-50 seconds (as shown in Figure~\ref{fig:simEndTimeTest}), each iteration of our simulation is 50 seconds long.

\begin{figure*}[!htbp]
    \centering
    \begin{subfigure}[t]{0.475\textwidth}
        \includegraphics[width=\textwidth]{simEndTime/SimEndTimeThroughput}
        \caption{Average network throughput}
        \label{fig:simEndTime1}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.475\textwidth}
        \includegraphics[width=\textwidth]{simEndTime/SimEndTimeDelay}
        \caption{Average end-to-end delay}
        \label{fig:simEndTime2}
    \end{subfigure}
    \caption{Average network throughput and end-to-end delay with varying simulation end time}
    \label{fig:simEndTimeTest}
\end{figure*}

We carefully set the tuning parameters of the proposed approach after numerous simulation trials. The channel switching probability of SU radios, \texttt{switchingProbability} was varied from 0.1 to 0.9 with a granularity of 0.05 and the reactivation probability of switched-off radios, \texttt{wakeUpProbability} was varied from 0.05 to 0.5 with a granularity of 0.05. Following these initial simulation results, we selected the value of these parameters that yielded best results in terms of throughput, delay, and packet drop ratio. In future, we aim to incorporate dynamic value selection of these parameters so that their values are updated on run time based on feedback from different performance metrics such as throughput, delay, and packet drop ratio. For the time being, we have fixed these parameters based on initial simulation results. The \texttt{switchingProbability} is set as 0.75 and the \texttt{wakeUpProbability} is set as 0.2. The channel sensing time for each of the cognitive radio is set as 0.01s while the channel switching time is set as 0.05s.

\section{Results and Analysis}

We start presenting our simulation results for a topology having 11 primary users and 24 secondary users. Here, we vary the application data rate from the source of a flow over secondary users from 1 Mbps to 32 Mbps. Fig.~\ref{fig:topology4T}, \ref{fig:topology4D}, and \ref{fig:topology4P} show the performance of several variants of our proposed approach and other existing approaches.

Fig.~\ref{fig:topology4T} depicts total network throughput for all the approaches in response to a variation in the number of radios for different application data rates. In most of the cases, our proposed approaches obtain significantly higher network throughput than the existing ones. Here, at lower data rates (1-8 Mbps), total network throughput increases with an increase in the number of radios. After reaching an optimal point, throughput starts degrading. At higher data rates (16 and 32 Mbps), the network throughput falls drastically from the single radio scenario and never again reaches the throughput obtained with single radio data transmission.

Fig.~\ref{fig:topology4D} illustrates that the feedback-based approaches experience significantly lower end-to-end delay than that achieved with the approach proposed by Zhong et al.~\cite{zhong2014capacity}. However, delay using our proposed approach is higher than that achieved with the approach proposed by Khan et al.~\cite{khan2015towards}. Here with our proposed approach, the delay becomes almost constant with an increase in the number of radios at lower application date rates (1-4Mbps). However, at higher data rates (8-32 Mbps), the delay rises with an increase in the number of radios per SU.

\input{topology4T}
\input{topology4D}
\input{topology4P}
\input{topology4PD}

Fig.~\ref{fig:topology4P} compares the average packet drop ratio of our proposed approaches against that of the existing approaches.  As illustrated in \cref{fig:topology4P}, the feedback-based approach achieves significantly lower packet drop ratios than all the existing ones. The feedback-based approach is also able to reduce the packet drop ratio significantly at lower data rates (1-8 Mbps) with the exploitation of multiple radios. However, at higher application data rates (16 and 32 Mbps), most of the packets get dropped resulting in high drop ratios. This explains why the network throughput at higher data rate does not improve even after the introduction of multiple data transmission radios.

Fig.~\ref{fig:topology4PD} shows the application layer packet delivery ratio of our proposed approaches against that of the existing approaches. Due to the efficient exploitation of multiple radios, our proposed approaches obtain significantly better packet delivery ratio than that achieved with the existing appraoches.

Table~\ref{tab:topology4RadioImprovement}, \ref{tab:topology4ChannelImprovement}, and \ref{tab:topology4RadioChannelImprovement} summarize average performance improvement using feedback-based approaches in comparison to the approaches proposed by Khan et al.,~\cite{khan2015towards} and Zhong et al.~\cite{zhong2014capacity}. The tables shows that the proposed approach outperforms the existing approaches in terms of all the performance metrics except end-to-end delay. In terms of total network throughput, the proposed approach obtains an average of 51\% improvement over the two existing approaches. Moreover, the proposed approach decreases packet drop ratio on an average 35\% and increases application layer packet delivery ratio on an average 32\% compared to existing approaches. Even though, the feedback-based approach experiences the higher delay in some cases, in average, the delay is improved by 13\% on an average.
%\input{topology4}

\input{topology4Improvement}

\section{Simulation Findings}

Though we have performed discrete event simulations for various network topologies varying the number of secondary users from 12 to 40 with a granularity of 4, in this thesis, due to space limitation, we have presented the simulation results for only one topology with 24 secondary users. Appendix~\ref{app:allTopologies} of this thesis contains the other results. Here, our proposed approach for MRCRNs obtains similar simulation results in case of other seven network topologies as well. Based on these simulation results, we obtain the following findings:

\begin{itemize}
    \item Over all these topologies, our proposed feedback-based approach improves total network throughput by 51\% on an average against that of existing approaches.
    \item Over all these topologies, our proposed feedback-based approach decreases packet drop ratio by 35\% on an average against that of existing approaches.
    \item Among three variants of our proposed feedback-based approach, radio channel feedback approach marginally (3\%) performs better over the other variants.
    \item For MRCRNs, our proposed feedback-based approach increases throughput with an increase in the number of radios for low to medium (1 -- 8Mbps) data rates. For high data rates (16 -- 32 Mbps), multiple radio introduction could not make significant impact on throughput and throughput usually degrades with an increase in the number of radios.
    \item For MRCRNs, our proposed feedback-based approach is able to make average end-to-end delay almost constant with an increase in the number of radios for low to medium (1 -- 8Mbps) data rates. For high data rates (16 -- 32 Mbps), delay usually increases with an increase in the number of radios.
    \item For MRCRNs, our proposed feedback-based approach improves average packet drop ratio with an increase in the number of radios for low to medium (1 -- 16Mbps) data rates. For high data rate (32 Mbps), packet drop ratio remains constant with an increase in the number of radios.
\end{itemize}

\endinput