• Home Page
• Documentation
• Examples
• Research Notes
• Small Simulation Studies
• Bayesian Network Repository

• About the Author

info & code

data & R code

data & R code

How to scale SHD and SID to a common scale, for instance, for plotting?

Causal discovery algorithms are commonly benchmarked using the Structural Hamming Distance (SHD) and the Structural Interventional Distance (SID). SHD measures structural differences between the equivalence classes of two networks and treats them as probabilistic graphical models. SID measures structural differences in terms of differential intervention outcomes. The question is: how to plot SHD and SID against each other in a way that makes visual sense across networks with very different numbers of nodes?

A simulation study based on tabu search and BIC ran over a wide range (37–724 nodes) of ireference networks from the Bayesian network repository suggests that SHD and SID can be rescaled to vary in a common range as

SHD / #ARCS ~ 1.5 * SID / (#ARCS / 2)^2

where #ARCS is the number of arcs in the reference network. The steps are, for each network:

1. Generate 20 data sets with sample size {0.1, 0.2, 0.5, 1, 2, 5} times the number of parameters in the network.
2. Perform structure learning on each data set.
3. Compute the SHD and the SID between the learned and reference networks.

Behold:

Even splitting the plot into different panels works well.

Note that the 1.5 multiplier is purely empirical, so it may not center SID with respect to SHD perfectly in every simulation study.

Repeating the same experiment with PC Stable confirms this and that the rescaling works across structure learning algorithms. This time, the empirical multiplier that aligns the points best around the diagonal is 1.

Moving to continuous data, the multipliers and the ranges of the normalized SHD and SID change. However, they still cluster adound the diagonal and fit in a square box near the origin.