Help with methodology

[u/LouhiVega]

Hello guys, I would like some guidance.

I have several methodologies to compare. Each one is non deterministic and everything is synthetic, performed in computer.

I can apply one methodology to a set of problems and see how it performs.

Each set has "n=10" conditions and "m=20" seeds. So, Each dataset consists in n x m elements. All equal among methodologies (I.e., condition is the same and the seed is the same between the methodologies).

Assuming that Each element of the dataset is a score in which lower is better, I would like to answers the following questions:

1. Which methodology is the best ? Is so, by how much ?

2. Which methodologies are equal statistically speaking ?

Thanks in advance.

[u/efrique]

Are these "seeds" random number seeds (start values for the RNG - in which case they would just represent replicates and their effect would be in the noise term) or something else (perhaps representing, say, different species of plant seeds), whose effect might be important, even if not of direct interest? Be explicit, don't make us guess and maybe guess wrong.

How is score obtained? Is it necessarily positive? Is it a count, a time, etc ...? Is there an upper bound on its value? Are any values missing or censored (e.g. if you're looking at times and one run takes such a very long time that you stop it, that's right censored at the stop time; all you know is that the total time would be at least that long)

Given each condition has an underlying distribution of scores, what aspect of the distribution do you want to compare to determine best? (means? medians? 99th percentiles? Conditional tail expectation? etc etc)

[u/LouhiVega → u/efrique]

Thank you for commenting.

Each methodology is a metaheuristic, each condition is a problem, each seed as a start value for the RNG. If I perform the same methodology, condition and seed, I must have the same outcome and the same path to it (i.e., each iteration is expected to be the same).

Briefly, exemplifying, each metaheuristic is like runner, which will run a given road. That path has obstacles, which would be a condition. Since the methodology is stochastic, I ask them to run several times and measure the performance.

Each condition and seed is equal between each methodology. Although a seed S1 in Method M1 can be lucky, and then find good solutions faster. While S1 in M2 can be unlucky, and then find good solutions way slower.

Each score is a metric regarding the performance of the metaheuristic computationally speaking. It can be problems solved per hour, problems solved at all, real time needed to solve. Each run has a limitation of 4 hours, and there are runs with such result. There are no missing values, although there are timeout situations (i.e., 4 hours). Each problem stops if it reaches either timeout or upper bound predefined result, in timeout situation it is expected that the upper bound was not reached.

In this case I have a metaheuristic (method), which performs better or worse depending on the condition. Since it is stochastic, I ran it under 10 seeds.

I want to compare these methods looking the full picture. For instance, if Method M1 has less problems solved at all considering all data set (conditions and seed at once) then it is possible to argue that M1 converges faster.

What I am confused is how to analyse one dataset. I mean, I have paired conditions but also seeds.

Please, if I am not clear enough let me know.

[u/Gray-Jay-]

At first glance, this looks like a factorial design with methodology and condition as factors, with the seeds providing replication. Testing the Methodology × Condition interaction would be of primary interest, since you indicate that relative method performance depends on condition.

If a given seed meant that each methodology was subjected to the same random inputs, then seed could be treated as a blocking factor in the factorial design.

If you consider time to solve the problem as your outcome, your data are right censored since you stop the experiment after 4 hrs. That suggests treating this as a factorial survival or time-to-event problem (possibly blocked) rather than analyzing completion times with ANOVA.

[u/LouhiVega → u/Gray-Jay-]

Ty for commenting. My concern is not actually how methodology interacts with conditions, because I just cannot change the methodology based on particular cases. I would like to know how the methodology handle it in the big picture, assuming all conditions and all seeds. Is it possible? How should I approach it?