Hi Kenneth, it looks like it is not supported yet. I got the following error message:

'create-table' (choose from 'run', 'available_tasks', 'available_benchmarks', 'create_meta')

Hi @lifu-tu this is a suggestion for a feature, not an implemented feature - I have made the main comment a bit more clear, but it sounds like it is something that you would like as well :)

@KennethEnevoldsen @Samoed, I have thought of following solutions here:
Use the similar implementation as of make_leaderboard.py and get scores. Based on it, we will create a comparison table and saved it on maybe at another argument called saved_path. Are we expecting this only or am I missing something here?

Yes. I'm also suggest to support multiple output format xlsx, csv, md etc. Also there should be benchmarks as optional and use local path for building the table

@Samoed , if benchmarks is optional then which results we will consider? Isn't that if benchmark is mentioned then, the tasks will be fixed because we consider only those tasks under that particular benchmark and take results of all models on those tasks only?

If benchmark specified then yes, otherwise all results can be taken from results dir

If benchmark specified then yes, otherwise all results can be taken from results dir

So, we will consider results of all tasks for that particular model, right?
Also, how should the comparison table be, just a single name with results of each mentioned models on different tasks, right?

So, we will consider results of all tasks for that particular model, right?

Yes

Also, how should the comparison table be, just a single name with results of each mentioned models on different tasks, right?

I think like

But I think that we can also set aggregation level for slit and subset

@Samoed What exactly subset is here? and can you clarify what exactly you mean by aggregation level for split and subset

Also, won't split be depend on task and not model, so it will be different for each task?

What exactly subset is here?

Tasks have subsets eval_langs which are subsets of task and all results have these subsets too.

Also, won't split be depend on task and not model, so it will be different for each task?

Yes

can you clarify what exactly you mean by aggregation level for split and subset

I think we can have parameter aggregation_level for this feature that will average scores for selected level

• split - no aggregation
• subset - aggregate subset results of different split. E.g. mean score of test and val subsets.
• task - aggregate results on subsets and splits

Also, won't split be depend on task and not model, so it will be different for each task?

Yes

So, the split field should be different for each task, then we should not have that split column, right?

Also, won't split be depend on task and not model, so it will be different for each task?

Yes

So, the split field should be different for each task, then we should not have that split column, right?

No, we should have split column if selected split aggregation

No, we should have split column if selected split aggregation

I am still not able to understand it clearly.
Are you saying that if aggregation is 'split' then for a given model, we will have subsets, and for each subsets we will have separate results of split, like following:

For subset aggregation level, we will take aggregate of different splits like dev, test so we will have table like:

For task aggregation level, we will take aggregate of different splits like dev, test so we will have table like:

Yes, like this!

@Samoed , I have few more doubts:

1. Is the subsets which you are talking about is hf_subsets only?
2. Are the subsets same for all tasks? Or is it possible that some tasks have some subsets while other have something different. Because if same subsets are not present for all tasks, then how we will be having split aggregation_level?
3. What exactly hierarchy is here? Is it that for a particular model, we have results of different tasks. For each tasks, we have results for different splits, and under each split we have results for different subsets?
4. Also, are the subsets same for different splits for a particular task?

1. Yes
2. Tasks have different subsets. Aggregation should aggregate results results inside all tasks. E. g. for each task mean of all task's splits or subsets
3. Yes, that right
4. No, subsets are different too

Can you once give me how table look like for subset and split aggregation_level. Because I think table that I have mentioned previously was incorrect as I assumed there that all tasks have same subsets.
Also, Is task aggregation_level table is correct and it is just same as what we have Performance per task table on leaderboard?

I see problem now. We can create multi-index (but it's a bit difficult to work) or transpose it.

For subset aggregation leve

For task aggregation level

Example of transpose table

@KennethEnevoldsen What do you think?

@Samoed

Is it that for a particular model, we have results of different tasks. For each tasks, we have results for different splits, and under each split we have results for different subsets?

If this is hierarchy, then for transpose tables, shouldn't we have in 1st table, 1st split column and then subset column. and also in 2nd table split column only

It's not directly transpose, but just changing axis. We have hierarchy task->subset->split and it should follow it

import pandas as pd
import numpy as np

arrays = [
    ["task 1"] * 4,
    ["subset 1"] * 2 + ["subset 2"] * 2,
    ["test", "dev"] * 2,
]
tuples = list(zip(*arrays))
index = pd.MultiIndex.from_tuples(tuples, names=["task", "subset", "split"])
df = pd.DataFrame(np.random.randn(4), index=index, columns=["model 1"])

                        model 1
task   subset   split
task 1 subset 1 test  -0.556665
                dev   -0.370966
       subset 2 test   0.107148
                dev    0.622765

We have hierarchy task->subset->split and it should follow it

Is it that for a particular model, we have results of different tasks. For each tasks, we have results for different splits, and under each split we have results for different subsets

You mentioned that it was task->split->subset when I asked this before. And also for any results files from results repo, I am seeing this hierarchy only.

You mentioned that it was task->split->subset when I asked this before.

Sorry for the confusion, but the correct order should be task -> subset -> split.

And also for any results files from results repo, I am seeing this hierarchy only.

Yes, but they have subsets inside

Yes, but they have subsets inside

so they are inside split only. How then hierarchy is: task -> subset -> split

@KennethEnevoldsen, I was working on this one. It will be good if you can have an opinion on the above discussion.

I think the suggestion by @Samoed proposed here is spot on.

So task -> subset -> split.

Anything I missed?

I think the suggestion by @Samoed proposed here is spot on.

Just wanted to confirm whether we should go with this or not?

So task -> subset -> split.

Just have confusion here in this hierarchy. In json files of any results, I am seeing hierarchy as:task->split->subset and I am referring to subset as hf_subset

Ah good point. Follow the JSON file:

task -> split -> subset

Ah good point. Follow the JSON file:

task -> split -> subset

If thats case, then here, the order column should also follow that? And in 1st table, we should have 1st split column and then subset column. and also in 2nd table split column only. So, it will be as follows:

I am renaming this as subset instead of split as we have results for each subset under each split:

Renaming this as split aggregation level, as we will take aggregate of different subsets for each split. Table will look like:

Basically I am defining aggregation level as level upto which we want to take aggregate results. Task means results aggregate upto each task which can be obtained by aggregating results of subsets over all splits. split aggregation level means, we will take aggregate upto split, means aggregate of all subsets under each split. and 'subset` aggregation level means we will not take any aggregate and show results for each subset under each split.

Let me know if this sounds correct

Sounds correct!