how to compare likelihoods of lambda models

[chun-he-316]

Hi,
I want to do a likelihood ratio test between two models using CAFE5, just like the function of lhtest in CAFE4, but I did not find the relative parameter that achieves this purpose. Can you tell me how to do this using CAFE5?. If CAFE5 does not have this function, can you tell me how to calculate the degrees of freedom so that I can do my own test?

I am looking forward to your answer. I would appreciate it very much if you could help.

Thank you!

[benfulton]

You can just use the “simulation” function to do what lhtest used to do (it ran a bunch of simulations and then fit models for you). We haven't actually been able to prove that they are chi-squared distributed, so the degrees of freedom don't come into it.

[chun-he-316]

Hi,
Thank you for your answer.
I want to compared models with different numbers of parameters. Firstly, I calculated the single, global lambda with error_model and get the lambda value 0.0024. Then I tried "cafe5 -i filtered.cafe.input.tsv -t tree.txt --pvalue 0.05 -eBase_error_model.txt -y separate_lambda.txt -s100 -l 0.0024 -o simulation" and obtained the simulation.txt and simulation_truth.txt. "separate_lambda.txt" is the treefile with multiple lambdas. But I do not know how to obtain LR that help me to judge if a multi-lambda model is better than a single-lambda model.
I am not sure if I used it wrong. Can you give me a complete example of how to compare two models with different numbers of lambda and judge which one is better?
I am looking forward to your answer. I would really appreciate it if you could help.
Thank you!

[benfulton]

You need to run enough simulations to be able to create a distribution of likelihoods ratios [i.e., LR=2×(score of global λ model – score of multi-λ model)] under the null hypothesis. 100 simulations is minimal, 1000 would be better.

[hunterkwalt]

Hi,

Thanks for your help! I just want to make sure I'm doing this right.

1.) Run CAFE5 on my data under both global and multi model
2.) Use resulting lambda values to generate simulated data for both global-lambda and multi-lambda model
3.) Run CAFE5 on these simulated datasets
4.) Obtain -lnl values that are produced during CAFE run on simulated datasets
5.) Create likelihood ratio distribution from these by plotting 2x(score of global model (from simulations) - multi model (from sim))
6.) See if observed likelihood ratio 2x(score of global (from real data) - score of multi (from real data)) falls within this distribution.

Also, can I use the same rationale to compare if a multi-lambda model fits better than another multi-lambda model? For example, test if a 3 lambda model is significantly better than a 2 lambda model, or a 4 lambda model is significantly better than a 3 lambda model, etc.

Thanks again!
Hunter

[benfulton]

Yes, that all looks correct.

[hunterkwalt]

Thanks again!

Hunter