Improving Business Performance with Machine Learning | Juan Jose Munoz | June 2024

Because we use unsupervised learning algorithms, there is no widely available standard for measuring accuracy, although we can use domain knowledge to validate groups.

A visual inspection of the groups reveals that some benchmark groups contain a mix of economy and luxury hotels, but this does not make business sense as the demand for hotels is fundamentally different.

You can scroll down to the data and see some differences, but can you find your own way to measure accuracy?

We want to create a function that measures the consistency of the recommended benchmark set across each feature. One way to do this is to calculate the variance of each feature in each set. For each cluster, we can calculate the average of the variance of each feature and then average the variances for each hotel cluster to get a total model score.

Our expertise tells us that to have a comparable benchmark set, we need to prioritise hotels of the same brand, possibly in the same market, same country and, if we use different markets or countries, the market demographics should be the same.

With that in mind, we want to highly penalize our measurements for variances in these features. To do this, we use a weighted average to calculate the variances for each benchmark set. We also output the variances for the primary and secondary features separately.

In summary, to create a measure of precision we need:

1. Calculate the variance of a categorical variableOne common approach is to use an “entropy-based” measure, where the more diverse a category is, the higher its entropy (variance).
2. Calculate the variance of a numeric variable: You can calculate the standard deviation or range (the difference between the maximum and minimum values), which measures the spread of the numerical data within each cluster.
3. Normalize the data: Normalize the variance scores for each category before applying the weights to ensure that no single feature dominates the weighted average due solely to differences in scale.
4. Applying weights to different metrics: Weight each type of variance based on its importance to the clustering logic.
5. Calculating the weighted average: Calculate the weighted average of the variance scores for each cluster.
6. Aggregate scores across clusters: The total score is the average of these weighted variance scores across all clusters or rows. A low average score indicates that the model effectively groups similar hotels together and minimizes variance within clusters.

from scipy.stats import entropy
from sklearn.preprocessing import MinMaxScaler
from collections import Counter
def categorical_variance(data):
"""
Calculate entropy for a categorical variable from a list.
A higher entropy value indicates datas with diverse classes.
A lower entropy value indicates a more homogeneous subset of data.
"""
# Count frequency of each unique value
value_counts = Counter(data)
total_count = sum(value_counts.values())
probabilities = [count / total_count for count in value_counts.values()]
return entropy(probabilities)
#set scoring weights giving higher weights to the most important features
scoring_weights = {"BRAND": 0.3,
"Room_count": 0.025,
"Market": 0.25,
"Country": 0.15,
"Market Tier": 0.15,
"HCLASS": 0.05,
"Demand": 0.025,
"Price range": 0.025,
"distance_to_airport": 0.025}
def calculate_weighted_variance(df, weights):
"""
Calculate the weighted variance score for clusters in the dataset
"""
# Initialize a DataFrame to store the variances
variance_df = pd.DataFrame()
# 1. Calculate variances for numerical features
numerical_features = ['Room_count', 'Demand', 'Price range', 'distance_to_airport']
for feature in numerical_features:
variance_df[f'{feature}'] = df[feature].apply(np.var)
# 2. Calculate entropy for categorical features
categorical_features = ['BRAND', 'Market','Country','Market Tier','HCLASS']
for feature in categorical_features:
variance_df[f'{feature}'] = df[feature].apply(categorical_variance)
# 3. Normalize the variance and entropy values
scaler = MinMaxScaler()
normalized_variances = pd.DataFrame(scaler.fit_transform(variance_df),
columns=variance_df.columns,
index=variance_df.index)
# 4. Compute weighted average
cat_weights = {f'{feature}': weights[f'{feature}'] for feature in categorical_features}
num_weights = {f'{feature}': weights[f'{feature}'] for feature in numerical_features}
cat_weighted_scores = normalized_variances[categorical_features].mul(cat_weights)
df['cat_weighted_variance_score'] = cat_weighted_scores.sum(axis=1)
num_weighted_scores = normalized_variances[numerical_features].mul(num_weights)
df['num_weighted_variance_score'] = num_weighted_scores.sum(axis=1)
return df['cat_weighted_variance_score'].mean(), df['num_weighted_variance_score'].mean()

To keep our code clean and to keep track of our experiments, let's also define a function to save the results of our experiments.

# define a function to store the results of our experiments
def model_score(data: pd.DataFrame,
weights: dict = scoring_weights,
model_name: str ="model_0"):
cat_score,num_score = calculate_weighted_variance(data,weights)
results ={"Model": model_name,
"Primary features score": cat_score,
"Secondary features score": num_score}
return results
model_0_score= model_score(results_model_0,scoring_weights)
model_0_score

Now that we have a baseline, let's see if we can improve our model.

Improve the model through experimentation

Up until now, when you run this code, you didn't need to know what was going on under the hood.

nns = NearestNeighbors()
nns.fit(data_scaled)
nns_results_model_0 = nns.kneighbors(data_scaled)[1]

To improve the model, we need to understand the model parameters and how to manipulate them to obtain a better benchmark set.

First, let's take a look at the Scikit Learn documentation and source code.

# the below is taken directly from scikit learn source
from sklearn.neighbors._base import KNeighborsMixin, NeighborsBase, RadiusNeighborsMixin
class NearestNeighbors_(KNeighborsMixin, RadiusNeighborsMixin, NeighborsBase):
"""Unsupervised learner for implementing neighbor searches.
Parameters
----------
n_neighbors : int, default=5
Number of neighbors to use by default for :meth:`kneighbors` queries.
radius : float, default=1.0
Range of parameter space to use by default for :meth:`radius_neighbors`
queries.
algorithm : {'auto', 'ball_tree', 'kd_tree', 'brute'}, default='auto'
Algorithm used to compute the nearest neighbors:
- 'ball_tree' will use :class:`BallTree`
- 'kd_tree' will use :class:`KDTree`
- 'brute' will use a brute-force search.
- 'auto' will attempt to decide the most appropriate algorithm
based on the values passed to :meth:`fit` method.
Note: fitting on sparse input will override the setting of
this parameter, using brute force.
leaf_size : int, default=30
Leaf size passed to BallTree or KDTree.  This can affect the
speed of the construction and query, as well as the memory
required to store the tree.  The optimal value depends on the
nature of the problem.
metric : str or callable, default='minkowski'
Metric to use for distance computation. Default is "minkowski", which
results in the standard Euclidean distance when p = 2. See the
documentation of `scipy.spatial.distance
<https://docs.scipy.org/doc/scipy/reference/spatial.distance.html>`_ and
the metrics listed in
:class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
values.
p : float (positive), default=2
Parameter for the Minkowski metric from
sklearn.metrics.pairwise.pairwise_distances. When p = 1, this is
equivalent to using manhattan_distance (l1), and euclidean_distance
(l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.
metric_params : dict, default=None
Additional keyword arguments for the metric function.
"""
def __init__(
self,
*,
n_neighbors=5,
radius=1.0,
algorithm="auto",
leaf_size=30,
metric="minkowski",
p=2,
metric_params=None,
n_jobs=None,
):
super().__init__(
n_neighbors=n_neighbors,
radius=radius,
algorithm=algorithm,
leaf_size=leaf_size,
metric=metric,
p=p,
metric_params=metric_params,
n_jobs=n_jobs,
)

There's a lot going on here.

of Nearestneighbor Classes inheritNeighborsBaseis a case class for nearest neighbor estimators. This class provides general functionality needed for nearest neighbor searches, e.g.

• n_neighbors (number of neighbors to use)
• Radius (for radius-based neighbor search)
• Algorithm (the algorithm used to calculate the nearest neighbors, e.g. “ball_tree”, “kd_tree”, “brute”, etc.)
• metric (the distance metric to use)
• metric_params (additional keyword arguments for metric functions)

of Nearestneighbor The class inherits fromKNeighborsMixin and RadiusNeighborsMixinThese mixin classes add specific proximity searching functionality. Nearestneighbor

• KNeighborsMixin It provides functionality to find a fixed number k of nearest neighbors of a point. It does this by finding the distance to the neighbors and their indices, and building a graph of connections between points based on each point's k neighbors.
• RadiusNeighborsMixin It is based on a radial neighborhood algorithm and finds all neighbors within a specified radius of a point. This method is useful in scenarios where the focus is on capturing all points within a meaningful distance threshold rather than a fixed number of points.

Based on our scenario, KNeighborsMixin provides the required functionality.

Before we can improve our model, we need to understand one important parameter: the distance metric.

The documentation states that the NearestNeighbor algorithm uses the “Minkowski” distance by default, and provides a reference to the SciPy API.

in scipy.spatial.distanceThere are two mathematical expressions for the Minkowski distance.

∑u−v∣p​=(i∑u−vi∣p) 1/p.

This formula calculates the pth root of the sum of the power differences of all the elements.

The second mathematical expression for the “Minkowski” distance is:

∑u−v∑p​=(i∑wi​(∣ui​−vi​∣p)) 1/p

This is very similar to the first one, but introduces weights wi Accentuate differences or de-emphasize certain dimensions. This is useful when certain features are more relevant than others. By default, the setting is[なし]In this case, all features are given the same weight of 1.0.

This is a great option for improving your model, as it allows you to pass domain knowledge to the model and highlight the similarities that are most relevant to the user.

If you look at the formula, you can see the parameters. pThis parameter affects the “path” the algorithm takes to calculate distance. By default, p=2, which represents the Euclidean distance.

Euclidean distance can be thought of as drawing a straight line between two points to calculate the distance. This is usually the shortest distance, but it is not always the best way to calculate distance, especially in high-dimensional spaces. For more information on why this is the case, see this great paper online: https://bib.dbvis.de/uploadedFiles/155.pdf

Another common value for p is 1, which represents the Manhattan distance. Think of it as the distance between two points measured along a grid-like path.

On the other hand, increasing p to infinity leads to the Chebyshev distance, which is defined as the maximum absolute difference between corresponding elements of the vectors.It is useful in scenarios where you want to ensure that no single feature changes significantly, as it essentially measures the worst-case difference.

Reading and understanding the documentation revealed several possible options for improving the model.

By default, n_neighbors is 5, but in the benchmark set, we compare each hotel to its three most similar hotels, so we should set n_neighbors = 4 (target hotel + 3 peer hotels).

nns_1= NearestNeighbors(n_neighbors=4)
nns_1.fit(data_scaled)
nns_1_results_model_1 = nns_1.kneighbors(data_scaled)[1]
results_model_1 = clean_results(nns_results=nns_1_results_model_1,
encoders=encoders,
data=data_clean)
model_1_score= model_score(results_model_1,scoring_weights,model_name="baseline_k_4")
model_1_score

Based on the documents, we can pass weights to the distance calculation to highlight the relationships between some features. Based on our domain knowledge, we identified the features to highlight (in this case, brand, market, country, and market tier).

# set up weights for distance calculation
weights_dict =  {"BRAND": 5,
"Room_count": 2,
"Market": 4,
"Country": 3,
"Market Tier": 3,
"HCLASS": 1.5,
"Demand": 1,
"Price range": 1,
"distance_to_airport": 1}
# Transform the wieghts dictionnary into a list by keeping the scaled data column order
weights = [ weights_dict[idx] for idx in list(scaler.get_feature_names_out())]
nns_2= NearestNeighbors(n_neighbors=4,metric_params={ 'w': weights})
nns_2.fit(data_scaled)
nns_2_results_model_2 = nns_2.kneighbors(data_scaled)[1]
results_model_2 = clean_results(nns_results=nns_2_results_model_2,
encoders=encoders,
data=data_clean)
model_2_score= model_score(results_model_2,scoring_weights,model_name="baseline_with_weights")
model_2_score

Passing domain knowledge to the model via weights significantly improved the score. Now let's test the impact of distance measures.

So far we've been using Euclidean distance. Let's see what happens if we use Manhattan distance instead.

nns_3= NearestNeighbors(n_neighbors=4,p=1,metric_params={ 'w': weights})
nns_3.fit(data_scaled)
nns_3_results_model_3 = nns_3.kneighbors(data_scaled)[1]
results_model_3 = clean_results(nns_results=nns_3_results_model_3,
encoders=encoders,
data=data_clean)
model_3_score= model_score(results_model_3,scoring_weights,model_name="Manhattan_with_weights")
model_3_score

We saw some nice improvements when we reduced p to 1. Let's see what happens as p approaches infinity.

To use the Chebyshev distance, change the metric parameter to: Chebyshev. The default sklearn Chebyshev metric does not have a weight parameter, to get around this you can define a custom one. weighted_chebyshev metric.

#  Define the custom weighted Chebyshev distance function
def weighted_chebyshev(u, v, w):
"""Calculate the weighted Chebyshev distance between two points."""
return np.max(w * np.abs(u - v))
nns_4 = NearestNeighbors(n_neighbors=4,metric=weighted_chebyshev,metric_params={ 'w': weights})
nns_4.fit(data_scaled)
nns_4_results_model_4 = nns_4.kneighbors(data_scaled)[1]
results_model_4 = clean_results(nns_results=nns_4_results_model_4,
encoders=encoders,
data=data_clean)
model_4_score= model_score(results_model_4,scoring_weights,model_name="Chebyshev_with_weights")
model_4_score

Through experimentation, we were able to reduce the variance scores of key features.

Let's visualize the results.

results_df = pd.DataFrame([model_0_score,model_1_score,model_2_score,model_3_score,model_4_score]).set_index("Model")
results_df.plot(kind='barh')

Using weighted Manhattan distance seems to give the most accurate set of benchmarks for our needs.

The final step before implementing the benchmark set is to look at the sets with the highest primary feature scores and identify what steps to take against them.

# Histogram of Primary features score
results_model_3["cat_weighted_variance_score"].plot(kind="hist")

exceptions = results_model_3[results_model_3["cat_weighted_variance_score"]>=0.4]
print(f" There are {exceptions.shape[0]} benchmark sets with significant variance across the primary features")

These 18 cases will need to be reviewed to ensure the benchmark set is appropriate.

As you can see, with just a few lines of code and some knowledge of proximity search, we were able to set up an internal benchmark set that we can then distribute and measure our hotel's KPIs against the benchmark set.

You don't always need to focus on cutting edge machine learning techniques to create value – often simple machine learning can create significant value.

What are some easy challenges in your business that can be easily tackled with machine learning?

World Bank. “World Development Indicators.” Retrieved June 11, 2024, https://datacatalog.worldbank.org/search/dataset/0038117

Aggarwal, CC, Hinneburg, A., Keim, DA (n.d.). On the surprising behavior of distance metrics in high-dimensional spaces. IBM TJ Watson Research Center and the Institute of Computer Science, University of Halle. Retrieved from https://bib.dbvis.de/uploadedFiles/155.pdf

SciPy v1.10.1 Manual. scipy.spatial.distance.minkowskiRetrieved June 11, 2024 from https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.minkowski.html

GeeksforGeeks. Haversine Formula to Find Distance Between Two Points on a Sphere. Retrieved June 11, 2024, from https://www.geeksforgeeks.org/haversine-formula-to-find-distance-between-two-points-on-a-sphere/.

scikit-learn.Neighbors module. Retrieved June 11, 2024 from https://scikit-learn.org/stable/modules/classes.html#module-sklearn.neighbors