Guidance on Key Parameters and Counterfactual

[cmottac]

Hi, I’m currently using the SyntheticControl wrapper as outlined in the documentation. However, I’m relatively new to pymc, and while I can replicate the examples successfully, I’m unsure about which parameters are most critical to adjust in order to obtain robust and meaningful results. Are there any specific aspects or best practices I should be particularly mindful of when running these analyses? For instance, are priors important to set and fine tune? Or do I need to play around some number of iterations?

In addition, I have a more specific question. I acknowledge that to get the average causal impact and average I can use az.summary(result.post_impact.mean("obs_ind")). In case I want the average value of the counterfactual, is it correct to use az.summary(result.post_pred.mean("obs_ind"))? My rationale: if I want an estimate of the relative causal effect that the treatment has on some metric, I think the right way would be to consider the ratio of the causal impact to the counterfactual for that metric.

Thank you in advance,
Carlo

[cmottac]

Also, do you have any reference that can help me understand the structure of the object result.post_impact.mean("obs_ind")... I am using it but I am not comfortable as I don't precisely know what I am doing.
thanks**2

[drbenvincent]

Hi @cmottac

I’m unsure about which parameters are most critical to adjust in order to obtain robust and meaningful results. Are there any specific aspects or best practices I should be particularly mindful of when running these analyses? For instance, are priors important to set and fine tune? Or do I need to play around some number of iterations?

So if you are new to the Bayesian approach, then I think the best resources would be books (there's a list on the pymc website) or possibly the examples page to get used to checking the MCMC sampling process is good. In short, if you don't get any convergence or divergence warnings then you are mostly likely good.

More samples generally means you'll get a more accurate estimation of the true posterior distribution of parameters given the data, but in most cases the default (1000) is acceptable, then ramp that up a bit if accuracy is really crucial.

The priors are perhaps a different story. The synthetic control uses the WeightedSumFitter model, so you've got a Dirichlet distribution over the weightings. The parameters of that are certainly relevant, depending what your prior beliefs are. Different values will embody different beliefs - for example many equally sized weights, or a few large weights and many near-zero weights.

As of now, there's no easy way for users to provide custom priors. Adding this functionality is on our roadmap #387

[drbenvincent]

In addition, I have a more specific question. I acknowledge that to get the average causal impact and average I can use az.summary(result.post_impact.mean("obs_ind")). In case I want the average value of the counterfactual, is it correct to use az.summary(result.post_pred.mean("obs_ind"))? My rationale: if I want an estimate of the relative causal effect that the treatment has on some metric, I think the right way would be to consider the ratio of the causal impact to the counterfactual for that metric.

I think it makes sense to calculate the causal impact as a percent change for example. You could plot the causal impact curve but rather an absolute units on the y-axis you could plot % change.

Whether it makes sense to average over time, I'm less clear on. If there is an upward trend (for example) then it maybe becomes less clear how to interpret that.

But you could also do the % cumulative causal impact for example, and look at the final value in the time series.

Just some thoughts.

[drbenvincent]

Also, do you have any reference that can help me understand the structure of the object result.post_impact.mean("obs_ind")... I am using it but I am not comfortable as I don't precisely know what I am doing.

So these are xarray objects. Basically just like numpy arrays, but with labelled dimensions. It can take a bit of time to get used to xarray, but they have good docs. Hopefully that helps?