Decision Tree Regression and Classification in Machine Learning

What is Decision Tree?
Decision Tree is used to solve Regression and classification problems via Tree based methods using Root nodes,edges, decision node and leaf with different technicques for a given data set.

Decision Tree algoritham is one of the most important algoritham in machine learning which can be used in ensamble technicque especially bagging and boosting technicques.

How Decision Tree will be formed to predict the output?

The following technicques will be used When the data set is given to identify the output of a given Regression or classification problem.
1.Entropy
2.Information gain
3.Gini impurity or Gini Index.

To form a tree the below form of tree will be used
1.Root Node
2.Edge
3.Decision Node
4.Leaf
The below pictorial representaion will explain in simple way to understand the structure of Decision Tree model.

Lets start with a example in order to understand how Decison Tree Model will work to predict ouput of given data set.

I will use the famous example given to students in most of the websites to understand easily. To begin with example, Let me start with what is entropy.

Entropy: Entropy is the first step of identify the purity of data set. Entropy is some what complex logic to identify the purity of the data set because this
ID3 algoritham would use log function to predict the purity of data set.
Data set is pure and Decision Tree is perfect when entropy value is 0 where as Data set is impure and identified Decision tree is invalid.
The below given formula is used to identify Entropy.

Information Gain: Information gain is also used to form a best Decision Tree model along with the Entropy. Information Gain will use entropy
values to find best fit of the Decisoon Tree with reduced entropy values. The below formula and examples would be used to identify the Best deceion node.
Step 1: Calculate entropy of the target.

Step 2: The dataset is then split on the different attributes. The entropy for each branch is calculated. Then it is added proportionally, to get total entropy for the split. The resulting entropy is subtracted from the entropy before the split. The result is the Information Gain, or decrease in entropy.

Step 3: Choose attribute with the largest information gain as the decision node, divide the dataset by its branches and repeat the same process on every branch.

Step 4a: A branch with entropy of 0 is a leaf node.

Step 4b: A branch with entropy more than 0 needs further splitting.

Step 5: The ID3 algorithm is run recursively on the non-leaf branches, until all data is classified.
Gini Impurity: Gini impurity is similar to Entropy.Formula is different to identfy the correct nodes. Gini impurity is better than Entropy because the computational speed is very high.