|
|
|
|
@@ -1,10 +1,174 @@
|
|
|
|
|
\chapter{Evaluation of existing Systems}
|
|
|
|
|
\chapter{Evaluation of existing Systems} \todo{rename to survey?}
|
|
|
|
|
\label{cha:evaluation_of_existing_systems}
|
|
|
|
|
|
|
|
|
|
This chapter deals with work around domain reputation scoring systems that has been released. While there exist different types of algorithms, only those that follow a similar approach are taken into account here: namely those that use passive DNS logs and machine learning to calculate the reputation score. \todo{why this two or three?}
|
|
|
|
|
|
|
|
|
|
\section{Evaluation Scheme}
|
|
|
|
|
\label{sec:evaluation_scheme}
|
|
|
|
|
|
|
|
|
|
For a comprehensive evaluation, all input and output as well as the exact implementations (and/or the corresponding parameters that have been used for the analysis) of the algorithm was needed. Unfortunately, none of the publications we are dealing with here have released any (raw) input data, specifically the passive DNS logs and the filter lists for the training set. Neither has any of the algorithm's actual implementation been published. For this reason the evaluation of the existing systems is focusing on the results that have individually been published. Most importantly the detection rate as well as the false positive rate. Another important fact for this overview is what data has actually been used for the training and classification and where the data has been obtained. Passive DNS logs may be collected in different stages of the DNS resolution and might, due to e.g. caching, lead to the extraction of different information. A resolver running on the users machine might obtain much more traffic and such benefit from e.g. time based patterns which are not possible at higher level DNS servers that are not able to collect that traffic because the response has been cached on resolvers in a lower (DNS-) hierarchy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Notos}
|
|
|
|
|
\label{sec:notos}
|
|
|
|
|
|
|
|
|
|
\subsection{General}
|
|
|
|
|
\label{subsec:notos_general}
|
|
|
|
|
|
|
|
|
|
\textit{Notos} has been published in 2010 by \fsAuthor{Antonakakis:2010:BDR:1929820.1929844} at the Georgia Institute of Technology. It has been introduced as ``being the first [system] to create a comprehensive dynamic reputation system around domain names'' \fsCite[Section 1]{Antonakakis:2010:BDR:1929820.1929844}. \textit{Notos} is based on observations that malicious use of DNS usually can be distinguished from legitimate, professionally provisioned DNS services by unique characteristics. Fraudulent activities therefore usually utilize techniques to evade security countermeasures \fsCite{Antonakakis:2010:BDR:1929820.1929844}. This approach is mainly using passive historical DNS information that was obtained on multiple recursive resolvers distributed accross the Internet. For building a model of how resources are typically used in legitimate and malcious applications, information about vicious ip addresses and domain names is collected from different sources like honeypots, malware analysis services and spam-traps. Using this model, new domains that have never been seen before can be dynamically assigned with a reputation score of how likely this new domain is involved in malicious activities. Malcious activities in the context of \textit{Notos} are roughly described as: ``if it [a domain] has been involved with botnet C\&C servers, spam campaigns, malware propagation, etc.'' \fsCite[Section 3]{Antonakakis:2010:BDR:1929820.1929844}
|
|
|
|
|
|
|
|
|
|
\textit{Notos} uses some basic terminology which is shortly introduced here:
|
|
|
|
|
\begin{itemize}
|
|
|
|
|
\item A domain \textit{d} consists of several substrings which are described in \nameref{subsec:domain_names}. Abbreviations used in the following Sections are: \\
|
|
|
|
|
\textbf{Top-level domain:} TLD, where \(TLD(d)\) is the top-level domain of \textit{d} \\
|
|
|
|
|
\textbf{Second-level domain:} \(2LD(d)\) being the second-level domain of domain \textit{d} \\
|
|
|
|
|
\textbf{Third-level domain: } \(3LD(d)\) containing the three rightmost substrings separated by period for \textit{d}
|
|
|
|
|
\item Given domain \(d\) \(Zone(d)\) describes the set of domains that include \textit{d} and all subdomains of \textit{d}
|
|
|
|
|
\item \(D = \{d_1, d_2, ..., d_m\}\) representing a set of domains and \(A(D)\) all IP addresses that, at any time, any domain \(d \in D\) resolved to
|
|
|
|
|
\item \(BGP(a)\) consists of all ip addresses that are residing in the same \gls{bgp} prefix than \textit{a}
|
|
|
|
|
\item Analogously, \(AS(a)\) as the set of IP addresses located in the same \gls{as} than \textit{a}
|
|
|
|
|
\end{itemize}
|
|
|
|
|
|
|
|
|
|
\subsection{Architecture}
|
|
|
|
|
\label{subsec:notos_architecture}
|
|
|
|
|
|
|
|
|
|
The main goal of \textit{Notos} is to assign a dynamic reputation score to domain names. Domains that are likely to be involved in malicious activities are tagged with a low reputation score, whereas legitimate Internet services are assigned with a high reputation score.
|
|
|
|
|
\textit{Notos'} primary source of information is a database that contains historical data about domains and resolved ip addresses. This database is built using DNS traffic from two recursive ISP DNS servers (RDNS) and pDNS logs collected by the Security Information Exchange (SIE) which covers authoritive name servers in North America and Europe. For building a list of known malicious domain names, several honeypots and spam-traps have been deployed. A large list of known good domains has been gathered from the top sites list on \textit{alexa.com} which ranks the most popular websites in several regions. These two lists are referred to as the \textit{knowledge base} and are used to train the reputation training model.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To assign a reputation score to a domain \textit{d}, the most current set of IP addresses \(A_{c}(d) = \left\{a_{i}\right\}_{i=1..m}\) to which \textit{d} points is first fetched. Afterwards the pDNS database is queried for several information for this domain \textit{d}. The \textit{Related Historic IPs (RHIPs)} is the set of all IP addresses that ever pointed to this domain. In case domain \textit{d} is a third-level domain, all IP addresses that pointed to the corresponding second-level domain are also included. See Chapter~\ref{subsec:domain_names} for more information on the structure of domain names. If \textit{d} is a second-level domain, then all IPs that are pointed to from any of the third-level subdomains are also added to the RHIPs. In the next step, the set of \textit{Related Historic Domains (RHDNs)} is queried and covers all domains that are related to the currently processed domain \textit{d}. Specifically, all domains which ever resolved to an IP address that is residing in any of the ASNs of those IPs that \textit{d} currently resolves to. \todo{understandable?}
|
|
|
|
|
|
|
|
|
|
There are three types of features extracted from the database for \textit{Notos} that are used for training the reputation model (quotation from \fsCite[Section 3.1]{Antonakakis:2010:BDR:1929820.1929844}):
|
|
|
|
|
|
|
|
|
|
\begin{quote}
|
|
|
|
|
\begin{enumerate}
|
|
|
|
|
\item \textbf{Network-based features:} The first group of statistical features is extracted from the set of RHIPs. We measure quantities such as the total number of IPs historically associated with \textit{d}, the diversity of their geographical location, the number of distinct autonomous systems (ASs) in which they reside, etc.
|
|
|
|
|
\item \textbf{Zone-based features:} The second group of features we extract are those from the RHDNs set. We measure the average length of domain names in RHDNs, the number of distinct TLDs, the occurrence frequency of different characters, etc.
|
|
|
|
|
\item \textbf{Evidence-based features:} The last set of features includes the measurement of quantities such as the number of distinct malware samples that contacted the domain \textit{d}, the number of malware samples that connected to any of the IPs pointed by \textit{d}, etc.
|
|
|
|
|
\end{enumerate}
|
|
|
|
|
\end{quote}
|
|
|
|
|
|
|
|
|
|
Figure~\ref{fig:notos_system_overview} shows the overall system architecture of \textit{Notos}. After all the features are extracted from the passive DNS database and prepared for further steps, the reputation engine is initialized. \textit{Notos'} reputation engine is operating in two modes. In offline mode, the reputation model is constructed for a set of domains using the feature set of each domain and the classification which can be calculated using the \textit{knowledge base} with black- and whitelist (also referred as training). This model can later be used in the online mode to dynamically assign a reputation score. In online mode, the same features that are used for the initial training are extracted for a new domain (resource record or RR, see Section~\nameref{subsubsec:dns_resource_records}) and \textit{Notos} queries the trained reputation engine for the dynamic reputation rating (see Figure~\ref{fig:notos_online_offline_mode}).
|
|
|
|
|
|
|
|
|
|
\todo{better explain EV, NM, DC}
|
|
|
|
|
\begin{figure}[!htbp]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[scale=.3, clip=true]{content/Evaluation_of_existing_Systems/Notos_System_overview.png}
|
|
|
|
|
\caption{Notos: System overview \fsCite[Figure 1]{Antonakakis:2010:BDR:1929820.1929844}}
|
|
|
|
|
\label{fig:notos_system_overview}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
\begin{figure}[!htbp]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[scale=.3, clip=true]{content/Evaluation_of_existing_Systems/Notos_offline-online_mode.png}
|
|
|
|
|
\caption{Notos: online and offline mode \fsCite[Figure 3]{Antonakakis:2010:BDR:1929820.1929844}}
|
|
|
|
|
\label{fig:notos_online_offline_mode}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Features}
|
|
|
|
|
\label{subsec:notos_features}
|
|
|
|
|
|
|
|
|
|
In this Section, all statistical features are listed and a short explanation, for what reason those have been chosen, is introduced.
|
|
|
|
|
|
|
|
|
|
The first group of features handles network-related keys. This group mostly describe how the owning operators of \textit{d} allocate network resources to achieve different goals. While most legitimate and professionally operated internet services feature have a rather stable network profile, malicious usage usually involves short living domain names and ip addresses with high agility to circumvent blacklisting and other simple types of resource blocking. Botnets usually contain machines in many different networks (\glspl{as} and \glspl{bgp}) operated by different organizations in different countries. Appropriate companies mostly acquire bigger ip blocks and such use consecutive IPs for their services in the same address space. This homogeneity also applies to other registration related information like registrars and registration dates. To measure this level of agility and homogeneity, eighteen statistical network-based features are extracted from the RHIPs (see Table~\ref{tab:notos_network-based_features}).
|
|
|
|
|
|
|
|
|
|
\begin{table}[!htbp]
|
|
|
|
|
\centering
|
|
|
|
|
\caption{Notos: Network-based features}
|
|
|
|
|
\label{tab:notos_network-based_features}
|
|
|
|
|
\begin{tabularx}{\textwidth}{|l|X|}
|
|
|
|
|
\hline
|
|
|
|
|
\textbf{Feature Source} & \textbf{Feature} \\ \hline
|
|
|
|
|
\multirow{9}{*}{\textit{BGP}} & \# of distinct BGP prefixes related to \(BGP(A(d))\) \\ \cline{2-2}
|
|
|
|
|
& \# of countries in which these BGP prefixes reside \\ \cline{2-2}
|
|
|
|
|
& \# of organizations that own these BGP prefixes \\ \cline{2-2}
|
|
|
|
|
& \# of distinct IP addresses in the sets \(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct IP addresses in the sets \(A_{2LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct BGP prefixes related to \(BGP(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct BGP prefixes related to \(BGP(A_{2LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of countries in which \(BGP(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of countries in which \(BGP(A_{2LD}(d)\) \\ \hline
|
|
|
|
|
\multirow{3}{*}{\textit{ASN}} & \# of distinct autonomous systems related to \(AS(A(d))\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct autonomous systems related to \(AS(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct autonomous systems related to \(AS(A_{2LD}(d)\) \\ \hline
|
|
|
|
|
\multirow{6}{*}{\textit{Registration}} & \# of distinct registrars associated with the IPs in the \(A(d)\) set \\ \cline{2-2}
|
|
|
|
|
& diversity in the registration dates related to the IPs in \(A(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct registrars associated with the IPs in the \(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of distinct registrars associated with the IPs in the \(A_{2LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& diversity in the registration dates for the IPs in \(A_{3LD}(d)\) \\ \cline{2-2}
|
|
|
|
|
& diversity in the registration dates for the IPs in \(A_{2LD}(d)\) \\ \hline
|
|
|
|
|
\end{tabularx}
|
|
|
|
|
\end{table}
|
|
|
|
|
|
|
|
|
|
The second group is about zone-based features and is extracted from the RHDNs. In contrast to the network-based features which compares characteristics of the historic IPs, the zone-based features handles characteristics of all historically involved domains. While legitimate services often involve many domains, they usually share similarities. ``For example, google.com, googlesyndication.com, googlewave.com, etc., are all related to Internet services provided by Google, and contain the string 'google' in their name.'' \fsCite[Section 3.2.2]{Antonakakis:2010:BDR:1929820.1929844}. In contrast, randomly generated domains used in spam campaigns are rarely sharing similarities. To calculate this level of diversity, seventeen features are extracted which can be found in Table~\ref{tab:notos_zone-based_features}:
|
|
|
|
|
|
|
|
|
|
\begin{table}[]
|
|
|
|
|
\centering
|
|
|
|
|
\caption{Notos: Zone-based features}
|
|
|
|
|
\label{tab:notos_zone-based_features}
|
|
|
|
|
\begin{tabularx}{\textwidth}{|l|X|}
|
|
|
|
|
\hline
|
|
|
|
|
\textbf{Feature Source} & \textbf{Feature} \\ \hline
|
|
|
|
|
\multirow{12}{*}{\textit{String}} & \# of distinct domain names in RHDNs \\ \cline{2-2}
|
|
|
|
|
& average \# of distinct domain names in RHDNs \\ \cline{2-2}
|
|
|
|
|
& standard deviation of \# of distinct domain names in RHDNs \\ \cline{2-2}
|
|
|
|
|
& mean of the occurrence frequency of each single character in the domain name strings in RHDNs \\ \cline{2-2}
|
|
|
|
|
& median of the occurrence frequency of each single character in the domain name strings in RHDNs \\ \cline{2-2}
|
|
|
|
|
& standard deviation of the occurrence frequency of each single character in the domain name strings in RHDNs \\ \cline{2-2}
|
|
|
|
|
& mean distribution of 2-grams (i.e. pairs of characters) \\ \cline{2-2}
|
|
|
|
|
& median distribution of 2-grams \\ \cline{2-2}
|
|
|
|
|
& standard deviation of 2-grams \\ \cline{2-2}
|
|
|
|
|
& mean distribution of 3-grams (i.e. triples of characters) \\ \cline{2-2}
|
|
|
|
|
& median distribution of 3-grams \\ \cline{2-2}
|
|
|
|
|
& standard deviation of the distribution of 3-grams \\ \hline
|
|
|
|
|
\multirow{5}{*}{\textit{TLD}} & \# of distinct TLD strings of each domain \(d_i\) in the RHDNs set \\ \cline{2-2}
|
|
|
|
|
& ratio between \# of domains \(d_i\) whose \(TLD(d_i)=".com"\) and the total \# of TLD different from ".com" \\ \cline{2-2}
|
|
|
|
|
& mean of the occurrence frequency of the TLD strings \\ \cline{2-2}
|
|
|
|
|
& median of the occurrence frequency of the TLD strings \\ \cline{2-2}
|
|
|
|
|
& standard deviation of the occurrence frequency of the TLD strings \\ \hline
|
|
|
|
|
\end{tabularx}
|
|
|
|
|
\end{table}
|
|
|
|
|
|
|
|
|
|
For the evidence-based features, public information and exclusively collected data from honeypots and spam-traps is collected. This \textit{knowledge base} primarily helps to discover if a domain \textit{d} is in some way interacting with known malicious IPs and domains. As domain names are much cheaper than ip addresses, malware authors tend to reuse IPs with updated domain names. The blacklist features detect the reuse of known malicious resources like IP addresses, \gls{bgp} prefixes and \glspl{as}.
|
|
|
|
|
|
|
|
|
|
\begin{table}[]
|
|
|
|
|
\centering
|
|
|
|
|
\caption{Notos: Evidence-based features}
|
|
|
|
|
\label{tab:notos_evidence-based_features}
|
|
|
|
|
\begin{tabularx}{\textwidth}{|l|X|}
|
|
|
|
|
\hline
|
|
|
|
|
\textbf{Feature Source} & \textbf{Feature} \\ \hline
|
|
|
|
|
\multirow{3}{*}{\textit{Honeypot}} & \# of distinct malware samples that, when executed, try to contact \(d\) or any IP address in \(A(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of malware samples that contact any IP address in \(BGP(A(d)\) \\ \cline{2-2}
|
|
|
|
|
& \# of samples that contact any IP address in \(AS(A(d))\) \\ \hline
|
|
|
|
|
\multirow{3}{*}{\textit{Blacklist}} & \# of IP addresses in \(A(d)\) that are listed in public IP blacklists \\ \cline{2-2}
|
|
|
|
|
& \# of IPs in \(BGP(A(d)\) that are listed in public IP blacklists \\ \cline{2-2}
|
|
|
|
|
& \# of IPs in \(AS(A(d))\) that are listed in public IP blacklists \\ \hline
|
|
|
|
|
\end{tabularx}
|
|
|
|
|
\end{table}
|
|
|
|
|
|
|
|
|
|
\todo{all formulas explained?}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{figure}[!htbp]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[scale=.3, clip=true]{content/Evaluation_of_existing_Systems/Notos_features.png}
|
|
|
|
|
\caption{Notos: Computing network-based, zone-based, evidence-based features \fsCite[Figure 2]{Antonakakis:2010:BDR:1929820.1929844}}
|
|
|
|
|
\label{fig:notos_features}
|
|
|
|
|
\end{figure}
|
|
|
|
|
\todo{not referenced atm}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Reputation Engine}
|
|
|
|
|
\label{subsec:notos_reputation_engine}
|
|
|
|
|
|
|
|
|
|
The reputation engine is used to dynamically assign a reputation score to a domain \textit{d}. In order to be able to achieve this, the engine has to be trained first. The training clusters
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
==> a low false positive rate (0.38\%) and high true positive rate (96.8\%).
|
|
|
|
|
|
|
|
|
|
\section{Exposure}
|
|
|
|
|
\label{sec:exposure}
|
|
|
|
|
|
|
|
|
|
|
