Tianchi YU(于天池)'s Blog

Performance Benefits From Underutilized CPU

Fri, 18 Aug 2023 00:00:00 GMT

Notes on CPU utilization and performance.

How to Write A CORRECT LLVM Pass As A Plugin

Sun, 06 Aug 2023 00:00:00 GMT

Notes on writing an LLVM pass as a plugin.

What is Three Address Code

Wed, 02 Aug 2023 00:00:00 GMT

Three Address Code (TAC) is an intermediate code generation technique used by compilers to represent high-level language constructs in a lower-level, more manageable form. TAC is called "three address code" because each instruction in this intermediate representation typically involves at most three addresses or operands. These addresses can represent variables, constants, or temporary results. TAC is an essential part of the compilation process as it simplifies the optimization and code generation phases.

Characteristics of Three Address Code

Simplicity: TAC uses a simple and limited set of instructions, making it easier to analyze and optimize.
Efficiency: TAC strikes a balance between high-level language constructs and low-level machine instructions, allowing for efficient code generation.
Flexibility: TAC can represent a wide range of programming constructs, including arithmetic operations, control flow statements, and function calls.
Ease of Optimization: The simplicity of TAC makes it easier for compilers to apply various optimization techniques to improve the performance of the generated code.

Basic Structure of TAC

A typical TAC instruction consists of at most three operands, which can be variables, constants, or temporary values. The general format of a TAC instruction is as follows:

result = operand1 operator operand2

Here, result is the variable or temporary location where the result of the operation is stored, operand1 and operand2 are the input operands, and operator is the operation to be performed (e.g., +, -, *, /).

Example

Consider the following high-level code snippet:

a = b + c * d;

The corresponding TAC representation might look like this:

t1 = c * d
a = b + t1

In this example, t1 is a temporary variable used to store the result of the multiplication before it is added to b.

Conclusion

Three Address Code is a valuable intermediate representation used by compilers to bridge the gap between high-level programming languages and low-level machine code. Its simplicity, efficiency, and flexibility make it an ideal choice for optimizing and generating code for a wide range of programming constructs. Understanding TAC is essential for anyone interested in compiler design and implementation.

Lexical Analysis, Syntax Analysis, and Semantic Analysis

Tue, 01 Aug 2023 00:00:00 GMT

Notes on lexical analysis, syntax analysis, and semantic analysis in compilers.

Rust Basics 1(Quick Review)

Sat, 01 Jul 2023 00:00:00 GMT

Rust Basics(1)

Anatomy of a Rust Program

indent with four spaces, not a tab
println! calls a Rust marco
end the line with a semicolon(;)

Compiling and Running Are Seperate Steps

Rust is an ahead-of-time compiled language.

Cargo

Cargo creats a project in a new directory and initializes a new Git repository along with a .gitignore file.

Note: Git is a common version control system. You can change cargo new to use a different version control 
system or no version control system by using the --vcs flag. Run cargo new --help to see the available 
options.

Cargo expects sources files to live inside the src directory and puts the binary in a directory named debug.

cargo new
cargo build
cargo run
cargo check

TOML(Tom's Obvious, Minimal Language)

It is Cargo's configuration format.

Building for Release

cargo build --release compile the project with optimizations.

Crate

A crate is a collection of Rust source code files. For example, the rand crate is a library crate, which contains code that is intended to be used in other programs.

external crates

The external crates are added into projects as dependencies. Cargo fetches the latest versions of everything that dependency needs from the registry, which is a copy of data from Crates.io.

Cargo.lock

If Cargo.lock exists, Cargo will use the versions specified there rather than doing all the work of figuring out versions again, which lets us have a reproducible build automatically.

update

cargo update

helps you to update a crate.

Common Programming Concepts

Variables and Mutability

By default, variables in Rust are immutable.

You would get compile-time errors when we attempt to change a value that is designated as immutable.

Constants

Like immutable variables, but constants:

don't accept mut syntax.
use const keyword
the type of the value must be annotated

Shadowing

fn main() {
    let x = 5;

    let x = x + 1;

    {
        let x = x * 2;
        println!("The value of x in the inner scope is: {x}");
    }

    println!("The value of x is: {x}");
}

When you run it:

$ cargo run
   Compiling variables v0.1.0 (file:///projects/variables)
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
     Running `target/debug/variables`
The value of x in the inner scope is: 12
The value of x is: 6

But shadowing:

we create a new variable
we can change the type of the value

Data Types

Rust is a statically typed language, it must know the types of all variables at compile time.

Scalar Types

Four primary scalar types : integers, floating-point numbers, Booleans and characters.

Integer Types

Integer Literals in Rust

Number literals	Example
Decimal	98_222
Hex	0xff
Octal	0o77
Binary	0b1111_0000
Byte (u8 only)	b'A'

Integer Overflow

In debug mode, Rust includes checks for integer overflow and cause the program to panic at runtime.
In release mode, Rust performs two's complement wrapping for the value, and the program won't panic.

Floating-point Types

Two primitive types : numbers with decimal points f32 and f64.

fn main() {
    let x = 2.0; // f64

    let y: f32 = 3.0; // f32
}

Boolean Type

true and false

Character Type

Four bytes in size and Unicode Scalar Value.

Compound Types

Tuple

a variety fo types
a fixed length
comma-separated list
access a tuple element by using a period (.) followed by the index
tuple without any values has a special name, unit : an empty value ()

fn main() {
    let tup: (i32, f64, u8) = (500, 6.4, 1);
}

fn main() {
    let tup = (500, 6.4, 1);

    let (x, y, z) = tup;

    println!("The value of y is: {y}");
}

Array

same type
a fixed lenght
comma-separated list
access elements by using indexing

fn main() {
    let a = [1, 2, 3, 4, 5];
    
    let a: [i32; 5] = [1, 2, 3, 4, 5];

    let a = [3; 5]; // equal to [3, 3, 3, 3, 3];

    let first = a[0];
    let second = a[1];

}

Arrays are useful when you want your data allocated on the stack rather than the heap.

Runtime error : invalid array element access:

use std::io;

fn main() {
    let a = [1, 2, 3, 4, 5];

    println!("Please enter an array index.");

    let mut index = String::new();

    io::stdin()
        .read_line(&mut index)
        .expect("Failed to read line");

    let index: usize = index
        .trim()
        .parse()
        .expect("Index entered was not a number");

    let element = a[index];

    println!("The value of the element at index {index} is: {element}");
}

The program resulted in a runtime error at the point of using an invalid value in the indexing operation. This check has to happen at runtime, especially in this case, because the compiler can’t possibly know what value a user will enter when they run the code later.

Functions

Functions are prevalent in Rust code.

Rust code uses snake case as the conventional style for function and variable names.

fn main() {
    println!("Hello, world!");

    another_function();
}

fn another_function() {
    println!("Another function.");
}

In function signatures, you must declare the type of each parameter. Requiring type annotations in function definitions means the compiler almost never needs you to use them elsewhere in the code to figure out what type you mean.

Statements and Expressions

Function bodies are made up of a series of statements optionally ending in an expression.

Statements are instructions that perform some action and do not return a value.
Expressions evaluate to a resultant value.

Expressions do not include ending semicolons. If you add a semicolon to the end of an expression, you turn it into a statement, and it will then not return a value.

Functions with Return Values

We must declare the return type after an arrow (->).

We need to have an expression as the return value. If not, there might be mismatched types error.

Control Flow

if Expressions

if condition {

} else {

}

Boolean is expected after if, Rust cannot automatically convert non-Boolean to Boolean types.

if condition {

} else if condition {

} ... {

} else {

}

Using too many else if expressions can clutter your code, so if you have more that one, you might wnat to refactor your code. For example by match.

Using if in a let Statement

Because if is an expression, it could be used on the right side of a let statement.

fn main() {
    let condition = true;
    let number = if condition { 5 } else { 6 };

    println!("The value of number is: {number}");
}

Repetition with Loops

Loop is an expression.

fn main() {
    loop {
        println!("again!");
    }
}

Returning Values from Loops

fn main() {
    let mut counter = 0;

    let result = loop {
        counter += 1;

        if counter == 10 {
            break counter * 2;
        }
    };

    println!("The result is {result}");
}

Loop Labels to Disambiguate Between Multiple Loops

We could specify a loop label so that we use with break or continue to specify that those keywords apply to the labeled loop instead of the innermost loop.

Note that, the break keyword cannot be followed by both a return value and a loop label at the same time. The break statement is used to exit a loop or a block early, and it can be optionally followed by an expression to provide a return value for the enclosing function or block. However, it cannot be combined with a loop label.

The labels should be in format like : 'lable.

Conditional Loops with while

fn main() {
    let mut number = 3;

    while number != 0 {
        println!("{number}!");

        number -= 1;
    }

    println!("LIFTOFF!!!");
}

Looping Through a Collection with for

fn main() {
    let a = [10, 20, 30, 40, 50];

    for element in a {
        println!("the value is: {element}");
    }
}

fn main() {
    for number in (1..4).rev() {
        println!("{number}!");
    }
    println!("LIFTOFF!!!");
}

DIS - Classification

Thu, 11 Jun 2020 00:00:00 GMT

<head> <script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ tex2jax: { skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'], inlineMath: [['$','$']] } }); </script> </head>

DIS - Classification

Clustering and Classificatioin

For inferring global models of data collections there exist two types of approaches: descriptive and predictive modeling. We illustrate the difference among them by an example.

We assume that a set of data items (or objects) with two attributes a1 and a2 is given. Assume the global model we are interested in is a classification (or as often said labeling) of the data items. In descriptive modeling we just know the data items, as indicated by the points in the 2-dimensional grid. A descriptive modeling technique, such as clustering, produces classes, which are not known in advance. For doing this it relies on some criteria that specify when two data items probably belong to the same class. Such a criteria is usually based on a similarity measure.

A predictive modeling technique, such as classification, starts from a given classification (or labeling) of data items. Using that classification of the dataset the classification method infers conditions on the properties of the data objects, that allow to predict the membership to a specific class. For example, the prediction could be based on a partitioning of the attribute values along each dimension, as shown in the figure on the right. There, first attribute a1 is partitioned into two intervals, and for each of the intervals a different partitioning of the attribute a2 is used to determine the regions corresponding to classes. Misclassifications may occur as seen in the example.

Given a dataset of objects described by attributes, build a model that assigns objects to a class (or label)

no class info -> descriptive model(clustering) -> describe classes based on similarity of attribute values

Given a dataset of objects described by attributes, build a model that assigns objects to a class

class info -> predictive model(classification) -> predict classes based on known attribute values

Classification Problem

Classification creates a global model, that is used for predicting the class label of unknown data. Since the classification function is learned from existing data, this approach is also called a supervised learning approach.

Classification is clearly useful in many decision problems, where for a given data item a decision is to be made (which depends on the class to which the data item belongs). Classification is also often called predictive analytics.

Input: set of objects with categorical/numerical attributes and one class label

Output: A model that returns the class label given the object attributes

Model is a function represented as rules, decision trees, formulae, neural networks

Classification belongs to supervised ML

Objects have class information

Basic Approach

Model is learnt from a set of objects with known labels: training set

The quality of the model is evaluated by comparing the predicted class labels with those from a set of objects with known labels: test set

Test set is independent of training set, otherwise over-fitting will occur

The model is applied to data with unknown labels: prediction

Example of Classification

Model test and Usage

Problem formulation

Problem: Given a database D with n data items described by d categorical/numerical attributes and one categorical attribute (class label C)

Find:(rules, decision tree,...)

A function f: $X^d -> C$

Such that: classifies accurately the items in the training set generalises well for the (unknown) items in the test set

Characteristics of Classification Methods

Predictive accuracy
Speed and scalability
- Time to build the model(usually more expensive)
- Time to use the model
- In memory vs. on disk proceesing
Robustness
- Handling noise, outliers and missing values
Interpretability [可解释性]
- Understanding the model and its decisions(black box) vs. white box
- Compactness of the model [简洁性]

As for clustering methods, we can also identify for classification methods a range of criteria to assess and compare the properties of different approaches.

Predictive accuracy is the natural main objective to optimize for a classifier. It characterizes how well the classifier performs its job. Often we encounter a trade off between predictive accuracy and the speed and scalability of the method. Methods that achieve high accuracy tend also to be more expensive. As for clustering, also for classification noise and outliers can pose additional problems affecting accuracy. Finally, a very important criterion is the interpretability of the model. In many concrete applications, it is critical that humans are able to understand based on which criteria a classifier takes a decision, e.g. for accountability. Imagine a case, where an assurance policy is refused based on the decision taken by a classifier, and the client would oppose in court. Only with human-interpretable methods the decision could be argued for. However, the most powerful classifiers today tend to produce models that are very hard to interpret for humans, as they represent very complex functions.

Q&A

If a classifier has 75% accuracy, it means that ...

A. It correctly classifies 75% of the data items in the training set

B. It correctly classifies 100% of the data items in the training set but only 75% in the test set

C. It correctly classifies 75% of the data items in the test set

D. It correctly classifies 75% of the unknown data items

Answer: [<font color: white> C </font>]

Decision Trees

Nodes are tests on a single attribute
Branches are attribute values
Leaves are marked with class labels

A standard type of classification function is a decision tree. In a basic decision tree, at each level one of the available attributes is used to partition the data set based on the different attribute values. At the leaf level of the decision tree, the values of the class label attribute are found. Thus, for a given data item with unknown class label attribute, by traversing the tree from the root to the leaf according to its data values, its class can be determined by choosing the class label found at the leaf level. Note that in different branches of the tree, different attributes may be used for classification.

A decision tree is constructed in a top-down manner, by recursively splitting the training set using conditions on the attributes. How these conditions are determined is one of the key questions for decision tree induction. After the decision tree construction, it may occur that at the leaf level the granularity is too fine, i.e., many leaves correspond to outliers in the data. Thus, in a second phase such leaves are identified and eliminated.

The key problem in constructing a decision tree is thus to determine the attributes that are used to partition the data set at each level of the decision tree.

Decision Tree Induction: Algorithm

Tree construction (top-down divide-and-conquer strategy)

At the beginning, all training samples belong to the root
Examples are partitioned recursively based on a selected “most discriminative” attribute
Discriminative power determined based on information gain (ID3/C4.5)

Partitioning stops if

All samples belong to the same class → assign the class label to the leaf
There are no attributes left → majority voting to assign the class label to the leaf
There are no samples left

The basic algorithm for decision tree induction proceeds in a greedy manner. First the set of all data objects are associated with the root. Among all attributes one is chosen to partition the set. The criterion that is applied to select the attribute is based on measuring the information gain that can be achieved, or how much uncertainty on the classification of the data objects is removed by the partitioning.

Three conditions can occur such that no further partitions can be performed:

(1) all data objects are in the same class, therefore further splitting makes no sense,

(2) no attributes are left which can be used to split. Still data objects from different classes can be in the leaf, then majority voting is applied.

(3) no data objects are left.

Example

Based on this approach for attribute selection, we can now illustrate the induction of the decision tree. In a first step, age is chosen for a split. The partition 31..40 contains after the split only instances from one class, the positive class, thus for this branch of the tree the induction terminates.

For the partition <= 30 we find that the student attribute is the best to be chosen for further splitting. Further splitting makes no more sense, as the two resulting partitions, after splitting by the student attribute, are consisting of either positive or negative instances only.

Similarly, for the partition >40 we find that credit rating gives the largest information gain. As before, further splitting is no more needed, as the resulting partitions contain only positive respectively negative instances.

Attribute Selection

Entropy

At a given branch in the tree, the set of samples S to be classified has P positive and N negative instances

The entropy of the set S is

$$ H(P,N) = -\frac{P}{P+N}log_2\frac{P}{P+N} - \frac{N}{P+N}log_2\frac{N}{P+N} $$

Note:

If P = 0 or N = 0, then H(P,N) = 0 -> no uncertainty

If P = N, then H(P,N) = 1 -> maximal uncertainty

The approach is based on an information-theoretic argument. Assuming that we have a binary category, i.e., two classes P and N to which objects in S have to be assigned, we can compute the amount of information required to determine the class, by H(P, N), the standard entropy measure, where P and N denote the cardinalities of the classes P and N. Given an attribute A that can be used for partitioning the data collection, we can calculate the amount of information needed to classify the data after the split according to attribute A has been performed. This value is obtained by calculating H(P, N) for each of the partitions and weighting these values by the probability that a data item belongs to the respective partition.

Example

We illustrate the attribute selection process now for our running example. Initially the data contains P = 9 positive instances and N = 5 negative instances. This results in an entropy of 0.94, i.e. 0.94 bits are required to decide the class of one instance. We compute next the entropies of all partitions that result from splitting all attributes. For example, if we split for Age, we obtain 3 partitions, each with a different distribution of positive and negative instances; and thus with different entropies.

Information Gain

The information gained by a split can thus be determined as the difference of the amount of information needed for correct classification before and after the split. Thus we calculate the reduction in uncertainty that is obtained by splitting according to attribute A and select among all possible attributes the one that leads to the highest reduction. For our example we can conclude that it is best to split on attribute age.

Next we compute the weighted sum of all entropies of the partitions in a split. The weights correspond to the probability of an instance falling into an element of the partition. Computing this for all attributes shows, that the attribute H results in the lowest entropy, i.e., leaves the lowest remaining uncertainty about the class membership of instances after the split.

Computation

Attribute A partitions S into $S_1, S_2, ... S_v$ Entropy of attribute A is $$ H(A) = \sum_{i=1}^v\frac{P_i+N_i}{P+N}H(P_i,N_i) $$

The information gain obtained by splitting S using A is $$ Gain(A) = H(P,N) - H(A) $$ Choose the biggest information gain as the splited one.

Q & A

Given the distribution of positive and negative samples for attributes A1 and A2, which is the best attribute for splitting?

A1	P	N
a	2	2
b	4	0

A2	P	N
x	3	1
y	3	1

Answer:[<font color:white>A1</font>]

Pruning [修剪]

A common problem in classification is that a classifier may overspecialize and capture noise and outliers in the data[噪声和离群值对数据的影响], rather than general properties. One possibility to limit overspecialization would be to stop the partitioning of tree nodes when some specific criteria is met (e.g., number of samples assigned to the leaf node). A possible criterion is to stop partitioning when the majority of remaining samples falls into one class. However, in general it is difficult to specify a suitable criterion a priori (e.g. choosing the right value of epsilon).

Another alternative is to first build the complete classification tree, and then, in a second phase, prune subtrees that do not contribute to an efficient classification. Different approaches can be applied to that end: heuristic approaches[启发式方法] can identify subtrees that do not contribute to the classification accuracy, and eliminate those. A more principled approach is the use of the minimum description length principle. (MDL).

The construction phase does not filter out noise → overfitting

Pruning strategies

Stop partitioning a node when large majority of samples is positive or negative, i.e., 𝑁/(𝑁+𝑃) or 𝑃/(𝑁+𝑃)>1−𝜀
Build the full tree, then replace nodes with leaves labelled with the majority class, if classification accuracy does not change
Apply Minimum Description Length (MDL) principle

分类算法： https://blog.csdn.net/china1000/article/details/48597469

Minimum Description Length Pruning

The MDL is based on the following consideration: if the effort in order to specify a class (the implicit description of the class extension through a decision tree) exceeds the effort to enumerate all class members (the explicit description of the class by enumerating its extension), then the subtree is over classifying and non-optimal. To measure the description cost a suitable metrics for the encoding cost, both for trees and data sets is required. For trees this can be done by suitably counting the various structural elements needed to encode the tree (#nodes, #test predicates, # arcs), whereas for explicit classification, it is sufficient to count the number of misclassifications that occur in a tree node.

补充：剪枝的目的是为了通过删除部分节点和子树，以避免“过度合适”/“过学习”，因为过学习将导致我们作出的假设的泛化能力过差。最小描述长度（MDL）准则表示，解释一组数据最好的理论，应该使下面这两项之和最下：

描述理论所需要的比特长度；
在理论的协助下，对数据编码所需要的比特长度。

而决策树中的MDL应当是使得训练样本的大多数数据符合这棵树，其他的样本作为例外编码，使得下面这两项最小：

编码决策树所需要的比特，它代表了猜想
编码例外实例所需要的比特

Let M1, M2, ..., Mn be a list of candidate models (i.e., trees). The best model is the one that minimizes $$ L(M) + L(D|M) $$ where

L(M) is the length of the description of the model in bits (#nodes, #leaves, #arcs ...)
L(D|M) is the is the length of the description of the data when encoded with the model in bits (#misclassifications)

Continuous Attributes

With continuous attributes it does not make sense to create a separate path in the decision tree for every possible attribute value. Instead, in such a case, a binary decision tree is constructed. Binary decisions can be specified both for continuous and categorical attributes. For continuous attributes, the binary split is performed by selecting a threshold that separates the instances in those that have a larger and a smaller value than the threshold. For categorical attributes, a subset of attribute values can be chosen that distinguishes the instances in two subsets.

With continuous attributes we cannot have a separate branch for each value

use binary decision trees

Binary decision trees

For continuous attributes A a split is defined by val(A) < X
For categorical attributes A a split is defined by a subset X $\subseteq$ domain(A)

Example

This example shows a dataset with both categorical and continuous attributes and a possible binary decision tree for such a dataset. The class label in the example is Risk.

Splitting Continuous Attributes

Approach

Sort the data according to attribute value
Determine the value of X which maximizes information gain by scanning through the data items

Only if the class label changes, a relevant decision point exists

When splitting the dataset using a continuous attribute, we need to determine which is the optimal value to split the dataset based on this attribute. To that end, first the set of attribute values is sorted. Then the class labels are traversed, and whenever it changes a possible split point is found (it can be shown that splitting where class labels do not change is provably sub-optimal). At these points the information gain needs to be computed.

Scalability of Continuous Attributes

Naive implementation

At each step the data set is split in subsets that are associated with a tree node

Problem

For evaluating which continuous attribute to split, data needs to be sorted according to these attributes
Becomes dominating cost

In a naïve implementation of the splitting process, we would keep all data in a single table. This would imply that we would have for traversing the attributes in order to resort that table every time an attribute is investigated. Therefore, a more efficient approach is needed.

To avoid repeated resorting of data, for every attribute a separate and presorted table is kept. Once a split is chosen, we find two different situation. For the table that keeps the attribute that was used in the split, the table needs just to be partitioned into two subtables, maintaining the order. For the other attributes we have to select the subtables corresponding to the instances of the two partitions that have been formed. To that end a temporary hash table is constructed that allows to associate to each dataitem its partition. Then the attribute table is scanned and partitioned using the hashtable to decide for each entry to which partition it belongs. Note that in this approach, for continuous attributes, the resulting tables are again sorted as the order is preserved from the original table.

Idea: Presorting of data and maintaining order throughout tree construction

Requires separate sorted attribute tables for each attribute

Updating attribute tables

Attribute used for split: splitting attribute table straightforward
Other attributes
- Build Hash Table once associating tuple identifiers (TIDs) of data items with partitions
- Select data from other attribute tables by scanning and probing the hash table

Example

In this example we demonstrate of how attribute tables are split, when a decision node is introduced in the decision tree. Since the split is based on attribute Age, the table for Age can simply be split into two subtables at the threshold value. For the Car Type table we use the temporary hash table indicating partition membership to separate it into two subtables.

Q & A

When splitting a continuous attribute, its values need to be sorted …

A. to avoid overfitting

B. to prune the data

C. to define a binary split condition

D. to accelerate tree induction

Answer:[<font color:white>C</font>]

Characteristics of Decision Tree Induction

We summarize here some of the major strengths and weaknesses of standard decision tree induction.

Decision trees advantages

The information theoretic criteria used to select the most discriminative attribute is an embedded feature selection
No data preparation is needed, such as normalisation of data
The best aspect of using trees for analytics is that they are easy to interpret and explain, while more sophisticated ML algorithms (ANN, SVM) are seen as black-boxes that do not “explain” the classification decisions they make

Decision trees drawbacks

They can be extremely sensitive to small perturbations in the data: a slight change can result in a drastically different tree.
They can easily overfit. This can be compensated by validation methods and pruning, but remains a problem.
They are not incremental. If new data is available, the existing tree cannot be incrementally modified, but the whole tree must be reconstructed from scratch

Strengths

Automatic feature selection
Minimal data preparation
Non-linear model
Easy to interpret and explain

Weaknesses

Sensitive to small perturbation in the data
Tend to overfit
No incremental updates

Decision Tree Induction: Properties

Model: flow-chart like tree structure

Score function: classification accuracy

Optimisation: top-down tree construction + pruning

Data Management: avoiding sorting during splits

Classification Algorithms

Decision trees is one of the best known and historically first examples of a classification approach. Many other methods have been devised in studied over tie. These include basic methods (we will see some examples later), ensemble methods (discussed in the following), support vector machines (a paradigm based on splitting the space through hyper-planes), and neural networks (which are attracting recently significant attention and are nowadays among the best performing classifiers if very large training sets are available).

Decision tree induction is a (well-known) example of a classification algorithm

Alternatives

Basic methods: Naïve Bayes, kNN, logistic regression, ..
Ensemble methods: random forest, gradient boosting, …
Support vector machines
Neural networks: CNN, rNN, LSTM, …

Ensemble Methods

One important development in decision trees was the introduction of the idea of ensemble methods. The basic principle is simple: instead of constructing a single model, many different models are constructed independently. Even if each model is not very expressive (weak learners) their combination can be powerful (strong learner). Different ensemble methods are distinguished by the type of approach they are based on. Ensemble methods, which we will discuss in the following, learn several models in parallel, and combine then their predictions by voting or averaging. Stacking methods use more sophisticated techniques to combine model outputs, based themselves on learning methods. Finally, boosting learn models in sequence. In each step the samples of the training data are reweighted depending on whether they have been correctly classified.

Idea

Take a collection of simple or weak learners
Combine their results to make a single, strong learner Types
Bagging: train learners in parallel on different samples of the data, then combine outputs through voting or averaging
Stacking: combine model outputs using a second-stage learner like linear regression[线性回归]
Boosting: train learners on the filtered output of other learners

Random Forests: Algorithm

Random forests are an ensemble method based on bagging. The principle is very simple: K different decision trees are learnt in parallel from different (independent) samples of the data, and the classification is derived from a majority vote of the predictions.

Learn K different decision trees from independent samples of the data (bagging); vote between different learners, so models should not be too similar

Aggregate output: majority vote

Why do Ensemble Methods Work?

Here we give the argument why ensemble methods work: even if the individual classifiers are not very good (e.g. make 35% errors in prediction) their aggregate will be very strong. For example, if we have 25 classifiers, the probability that a majority of them, namely at least 13, make a wrong prediction is very small, namely 6%. In general, ensemble methods work well, if the individual models are better than random guessing. The figure illustrates the relation between the classification errors of individual classifiers and the aggregate classification accuracy.

Sampling Strategies

Two sampling strategies

Sampling data

select a subset of the data → Each tree is trained on different data

Sampling attributes

select a subset of attributes → corresponding nodes in different trees (usually) don’t use the same feature to split

For random forests the main issue is the choice of the sampling strategy, i.e., the generation of samples that are used for learning the individual models. Specifically, it consists of two different sampling strategy. The first, sampling data selects from the original dataset a sample. Thus each decision tree is trained on a different sample of data. The second, sampling attributes selects from the attributes available to take a decision a random subset. Thus even if a tree would have been constructed in the same way up to a level, the continuation might become different due to attribute sampling (e.g. the optimal attribute in one tree is not available for splitting in the other tree).

Algorithm context

Draw K bootstrap samples of size N from original dataset, with replacement (bootstrapping)
While constructing the decision tree, select a random set of m attributes out of the p attributes available to infer split (feature bagging)

Typical parameters

m ≈ sqrt(p), or smaller
K ≈ 500

Illustration of Random Forests

Random forests allow to learn much more complex functions than basic decision trees. This fact is illustrated in this visualization. For the same training data set different numbers of decision trees are constructed (rCART is a variant of decision trees). We observe that with increasing numbers of trees the decision boundaries become increasingly more complex and smoother, and thus a better separation among different classes can be achieved.

Q & A

The computational cost for constructing a RF with K as compared to constructing K decision trees on the same data

A. is identical

B. is on average larger

C. is on average smaller

Answer:[<font color:white>C</font>]

Characteristics of Random Forests

Random forests are a very popular method for classification due to the many advantages they offer. They are considered as the method of choice in cases where the data is dense, which means that the number of features is relatively low (in the thousands). Sparse data would be, for example, vector space representation of documents with very large vocabularies. In such cases, before applying a method such as random forests, a dimensionality reduction would have to be applied. This could be accomplished in the case of documents by creating a word embedding.

Strengths

Ensembles can model extremely complex decision boundaries without overfitting
Probably the most popular classifier for dense data (≤ a few thousand features)
Easy to implement (train a lot of trees)
Parallelizes easily, good match for MapReduce

More recently, in cases where large training sets are available or number of features is very large, deep neural networks exhibit better performance than random forests.

Weaknesses

Deep Neural Networks generally do better
Needs many passes over the data – at least the max depth of the trees
Relatively easy to overfit – hard to balance accuracy/fit tradeoff

references

Textbook

Jiawei Han, Data Mining: concepts and techniques, Morgan Kaufman, 2000, ISBN 1-55860-489-8 References
Leo Breiman (2001) “Random Forests” Machine Learning, 45, 5-32.
Shafer, John, Rakesh Agrawal, and Manish Mehta. "SPRINT: A scalable parallel classifier for data mining." Proc. 1996 Int. Conf. Very Large Data Bases. 1996.

Software Security 1 (Basic Principles)

Wed, 19 Feb 2020 00:00:00 GMT

Software Security 1 (Basic Principles)

Mathias Payer EPFL Spring 2020

Allow intended use of software, prevent unintended use that may cause harm.

Security principles / CIA security triad

Confidentiality: an attacker cannot recover protected data
Integrity: an attacker cannot modify protected data
Availability: an attacker cannot stop/hinder computation

Accountability/non-repudiation may be used as fourth fundamental concept. It prevents denial of message transmission or receipt.

Attacks and Defenses

Attack (threat) models
- A class of attacks that you want to stop
- What is the attacker’s capability?
- What is impact of an attack?
- What attacks are out-of-scope?
Defenses address a certain attack/threat model
- General: e.g., stop memory corruptions
- Very specific: e.g., stop overwriting a return address

Threat model

The threat model defines the abilities and resources of the attacker. Threat models enable structured reasoning about the attack surface.

Awareness of entry points (and associated threats)
Look at systems from an attacker’s perspective
- Decompose application: identify structure
- Determine and rank threats
- Determine counter measures and mitigation

Further reading: https://www.owasp.org/index.php/Application_Threat_Modeling

Threat model for a safe/lockbox

You want to protect your valuables by locking them in a safe.

In trust land, you don’t need to lock your safe. An attacker may pick your lock. An attacker may use a torch to open your safe. An attacker may use advanced technology (x-ray) to open it. An attacker may get access (or copy) your key.

Threat model: operating systems

Malicious extension【恶意扩展】: attacker-controlled driver injected in OS;在操作系统中加入攻击者控制的驱动程序\ Bootkit: compromise the boot process (BIOS, boot sectors)危及boot进程;\ Memory corruption: software bugs such as spatial and temporal memory safety errors or hardware bugs such as rowhammer;\ Data leakage【数据泄露】: the OS accidentally returns confidential data (e.g., randomization secrets);\ Concurrency bugs【并发问题】: unsynchronized【不同步】 reads across privilege levels result in TOCTTOU (time of check to time of use【检查时间到使用时间】) bugs;\ Side channels: indirect data leaks through shared resources such as hardware (e.g., caches), speculation (Spectre or Meltdown), or software (page deduplication);\ Malicious peripherals【恶意外围设备】: an attacker may connect malicious peripherals to an exposed bus;\ Resource depletion and deadlocks: stop legitimate computation【合法计算】 by exhausting【耗尽】 or blocking【锁死】 access to resources

Cost of security

There is no free lunch, security incurs overhead. Security is. . . expensive to develop, expensive to maintain, may have performance overhead, may be inconvenient to users.

Fundamental security mechanisms

Isolation

Isolate two components from each other. One component cannot access data/code of the other component except through a well-defined API.

Least privilege

The principle of least privilege ensures that a component has the least privileges needed to function.

最小特权原则是系统安全中最基本的原则之一。所谓最小特权(Least Privilege)，指的是"在完成某种操作时所赋予网络中每个主体(用户或进程)必不可少的特权"。最小特权原则，则是指"应限定网络中每个主体所必须的最小特权，确保可能的事故、错误、网络部件的篡改等原因造成的损失最小"。

Any further removed privilege reduces functionality Any added privilege will not increase functionality (according to the specification) This property constrains an attacker in the obtainable privileges

最小特权原则一方面给予主体"必不可少"的特权，这就保证了所有的主体都能在所赋予的特权之下完成所需要完成的任务或操作;另一方面，它只给予主体"必不可少"的特权，这就限制了每个主体所能进行的操作。最小特权原则要求每个用户和程序在操作时应当使用尽可能少的特权，而角色允许主体以参与某特定工作所需要的最小特权去签入(Sign)系统。被授权拥有强力角色(Powerful Roles)的主体，不需要动辄运用到其所有的特权，只有在那些特权有实际需求时，主体才去运用它们。如此一来，将可减少由于不注意的错误或是侵入者假装合法主体所造成的损坏发生，限制了事故、错误或攻击带来的危害。它还减少了特权程序之间潜在的相互作用，从而使对特权无意的、没必要的或不适当的使用不太可能发生。这种想法还可以引申到程序内部：只有程序中需要那些特权的最小部分才拥有特权。

Fault compartments 【故障舱】

Separate individual components into smallest functional entity possible. These units contain faults to individual components. Allows abstraction and permission checks at boundaries.

Note that this property builds on least privilege and isolation. Both properties are most effective in combination: many small components that are running and interacting with least privileges.

Example: mail server

Mail Transfer Agents (MTA) need to do a plethora of tasks:

Send/receive data from the network

Manage a pool of received/unsent messages

Provide access to stored messages for each user

-> Two approaches: sendmail and qmail

Sendmail uses a typical Unix approach with a large monolithic【庞大的】 server and is known for the high complexity and previous security vulnerabilities【脆弱点】.

QMail uses a modern least privilege approach with a set of communicating processes.

QMail [详见ppt]

qmail is a mail transfer agent (MTA) that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program. Originally license-free software, qmail's source code was later dedicated in the public domain by the author. When first published, qmail was the first security-aware mail transport agent; since then, other security-aware MTAs have been published. The most popular predecessor to qmail, Sendmail, was not designed with security as a goal, and as a result has been a perennial target for attackers. In contrast to sendmail, qmail has a modular architecture composed of mutually untrusting components; for instance, the SMTP listener component of qmail runs with different credentials from the queue manager or the SMTP sender. qmail was also implemented with a security-aware replacement to the C standard library, and as a result has not been vulnerable to stack and heap overflows, format string attacks, or temporary file race conditions.

Separate modules run under separate user IDs (isolation) Each user ID has only limited access to a subset of the resources (least privilege) Only one very small component runs as suid root Only one very small component running as root

SUID (Set owner User ID up on execution) is a special type of file permissions given to a file. Normally in Linux/Unix when a program runs, it inherit’s access permissions from the logged in user. SUID is defined as giving temporary permissions to a user to run a program/file with the permissions of the file owner rather that the user who runs it. In simple words users will get file owner’s permissions as well as owner UID and GID when executing a file/program/command.

Trust and correctness

Specific components are assumed to be trusted or correct according to a specification

Formal verification ensures that a component correctly implements a given specification and can therefore be trusted. Note that this property is an ideal property that cannot generally be achieved.

Hardware and software abstractions

Operating System (OS) abstractions

Provides process abstraction
Well-defined API to access hardware resources
Enforces mutual exclusion to resources
Enforces access permissions for resources
Restrictions based on user/group/ACL
Restricts attacker

OS process isolation:

Memory protection :protect the memory (code and data such as heap, stack, or globals) of one process from other processes;
Address space: working memory of one process

Today’s system implement address spaces (virtual memory) through page tables with the help of an Memory Management Unit (MMU)

Single domain OS (Hardware -> Application)
1. A single layer, no isolation or compartmentalization
2. All code runs in the same domain: the application can directly call into operating system drivers
3. High performance, often used in embedded systems
Monolithic OS(Hardware -> ->several applications)
1. Two layers: the operating system and applications
2. The OS manages resources and orchestrates access
3. Applications are unprivileged, must request access from the OS
4. Linux fully and Windows mostly follows this approach for performance (isolating individual components is expensive)
Micro-kernel
1. Many layers: each component is a separate process
2. Only essential parts are privileged
  1. Process abstraction (address spaces)
  2. Process management (scheduling)
  3. Process communication (IPC)
3. Applications request access from different OS processes
Library OS
1. Few thin layers; flat structure
2. Micro-kernel exposes bare OS services
3. Each application brings all necessary OS components

Hardware abstractions

Virtual memory through MMU/OS

Only OS has access to raw physical memory

DMA for trusted devices

ISA enforces privilege abstraction (ring 0/3 on x86)

Hardware abstractions are fundamental for performance

Access control

Authentication【证明】: Who are you (what you know, have, or are)?
Authorization【授权】: Who has access to object?
Audit/Provenance【审查，来源】: I’ll check what you did.

Authentication

There are three fundamental types of identification:

What you know: username / password
What you are: biometrics
What you have: smartcard / phone

Identification is stronger in combination:

two factors are better than one
Check for username/password and presence of a security token

Authorization

—— Information Flow Control (IFC)

Who can access what information?

Access policies are called access control models.
Access control models originally developed by the US military
- Users with different clearance levels【许可级别】 on a single system
- Data was shared across different levels of clearance
- Therefore the name “multi-level security”

—— Types of access control

Mandatory Access Control (MAC) 【强制/命令访问控制】 Idea: rule and lattice-based policy\
- Centrally controlled
- One entity controls what permissions are given
- Users cannot change policy themselves
Examples: Bell/LaPadula and Biba\

Bell-LaPadula模型 (BLP) 是一种状态机模型，用于在政府和军事应用中实施访问控制。它是由David Elliott Bell 和Leonard J. LaPadula在Roger R. Schell的指导下设计的，用于规范美国国防部 (DoD) 的多级安全 (MLS) 策略。该模型是计算机安全策略的形式状态转换模型，它利用访问主体和访问对象的安全等级来描述一系列访问控制规则。安全等级的范围从最敏感（如“最高机密”）到最不敏感（如“未分类”或“公开”）。Bell-LaPadula模型是保护与安全之间没有明显区别的模型之一。\ 在存在受信任主体的情况下，Bell-LaPadula模型可能会产生从高机密文档到低机密文档的信息流动。受信任主体不受*属性的限制，但必须证明其在安全策略方面是值得信任的。该安全模型针对访问控制，并被描述为：“下读，上写”。\ 在Bell-LaPadula模型中，用户只能在其自己的安全级别或更高的安全级别上创建内容（如，秘密研究人员可以创建秘密或绝密文件，但不能创建公共文件；不能下写）。相反，用户只能查看在其自己的安全级别或更低的安全级别的内容（如，秘密研究人员可以查看公共或秘密文件，但不能查看绝密文件；不能上读）。\

Multi level security model that enforces information flow control;\ All security levels are monotonically ordered;\ -> Objects (files) have a clearance level【许可级别】\ A given clearance allows reading objects of lower or equal clearance and writing files of equal or higher clearance;\ Summarized as read-down, write-up;\ Bell/LaPadula enforces confidentiality;\

注重机密性，此模型强调就是保证机密不能泄密。所以，机密程度越高，越保密，低的就不能读。写方面，高不能往低写，写的话机密的东西流到机密低的地方了。所以他不允许数据由高级往低级流，但允许低级往高级流，机密性低的数据，跑到机密性高的地方，并不破坏机密性。比如，1，2，3，4 ， 1级为最高机密，那么2级的人，不可以读1级数据，这将破坏保密原则。所以不能向上读。反之，1级的人，向2级区域写东西，则也会破坏机密性。

If implemented naively, attacker may overwrite confidential files. Mathias Payer

Biba模型

Multi level security model that enforces information flow control;\ All security levels are monotonically ordered;\ Objects (files) have an integrity level【完整性级别】\ A given clearance allows reading files of higher or equal clearance and writing files of lower or equal clearance;\ Biba introduces notion of integrity, separate from secrecy;\ Summary: read-up, write-down;\ Biba enforces integrity;\

完整性1，2，3，4级， 1级是最高完整性，比如是董事会最高决议，它并不需要保密，谁都能看，但任何人不得破坏其完整性，不能修改其决议，只有1级的人才能修改。所以2级的人不可以写1级的内容。但2级可以读1级的内容，来生成2级的信息。此时，数据只能由高向低流，而禁止从低向高流。

If implemented naively, attacker may leak privileged information.

另，状态机模型的介绍另见笔记“有限状态机”。

Discretionary【任意的，无条件的】 Access Control (DAC) 【自由访问控制】

Idea: object owner specifies policy

MAC requires central control, empower【许可，准许】 the user!
User has authority over her resources (introduce ownership)
User sets permissions for her data if other users want access
For example: Unix permissions

Can become complex and may require per-file special casing.

Role-Based Access Control (RBAC) 【基于角色访问控制】

Policy defined in terms of roles (sets of permissions), individuals are assigned roles, roles are authorized for tasks.

Access permission is broken into sets of roles.
Users get assigned specific roles
Administration privileges may be a role.

Roles simplify special casing but roles can become overly complex.

Access control matrix

map access rights for subjects to objects

	foo	bar	baz
user	rwx	rw	rw
group	rx	r	r
other	rx		r

Used, e.g., for Unix/Linux file systems or Android/iOS/Java security model for privileged APIs.

Introduced by Butler Lampson in 1971 http://research.microsoft.com/enus/um/people/blampson/08-Protection/Acrobat.pdf

Summary

Software security goal: allow intended use of software, prevent unintended use that may cause harm.
Security triad, core principles: Confidentiality, Integrity, Availability.
Security of a system depends on its threat model.
Concepts: isolation, least privilege, fault compartments, trust.
Security relies on abstractions to reduce complexity.
Reading assignment: Butler Lampson, Protection http://doi.acm.org/10.1145/775265.775268