Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error Exporting Ensembl 111 #25

Closed
ACastanza opened this issue Feb 1, 2024 · 6 comments
Closed

Error Exporting Ensembl 111 #25

ACastanza opened this issue Feb 1, 2024 · 6 comments

Comments

@ACastanza
Copy link
Contributor 


INFO:root:exporting ensembl genes to output/homo_sapiens_core_111_38: version 111
INFO:root:connection_url: mysql+mysqlconnector://[email protected]:3306/homo_sapiens_core_111_38
INFO:root:exporting genes data
INFO:root:Ran 'genes' query returning 70,711 rows. Head:

   ensembl_gene_id  ensembl_gene_version  ... seq_region_strand primary_assembly
0  ENSG00000000003                    16  ...                -1             True
1  ENSG00000000005                     6  ...                 1             True
2  ENSG00000000419                    14  ...                -1             True
3  ENSG00000000457                    14  ...                -1             True

[4 rows x 19 columns]
INFO:root:Ran 'gene_xrefs' query returning 505,967 rows. Head:

   ensembl_gene_id   xref_source  ... xref_info_type xref_linkage_annotation
0  ENSG00000000003  ArrayExpress  ...         DIRECT                    None
1  ENSG00000000003    EntrezGene  ...      DEPENDENT                    None
2  ENSG00000000003     GeneCards  ...      DEPENDENT                    None
3  ENSG00000000003          HGNC  ...         DIRECT                    None

[4 rows x 7 columns]
INFO:root:Ran 'gene_alt_alleles' query returning 14,511 rows. Head:

   ensembl_gene_id  ...  ensembl_created_date
0  ENSG00000282572  ...   2015-06-01 18:57:05
1  ENSG00000281951  ...   2015-06-01 18:57:05
2  ENSG00000282572  ...   2015-06-01 18:57:05
3  ENSG00000273644  ...   2014-06-09 10:49:07

[4 rows x 7 columns]
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/commands.py:39   │
│ in export_all                                                                │
│                                                                              │
│   36 │   def export_all(species: str = "human", release: str = "latest") ->  │
│   37 │   │   """Export datasets and then notebooks."""                       │
│   38 │   │   # Cannot use a classmethod here <https://github.com/related-sci │
│ ❱ 39 │   │   Commands.export_datasets(species=species, release=release)      │
│   40 │   │   Commands.export_notebooks(species=species, release=release)     │
│   41 │                                                                       │
│   42 │   @staticmethod                                                       │
│                                                                              │
│ ╭───── locals ──────╮                                                        │
│ │ release = '111'   │                                                        │
│ │ species = 'human' │                                                        │
│ ╰───────────────────╯                                                        │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/commands.py:24   │
│ in export_datasets                                                           │
│                                                                              │
│   21 │   │   │   f"exporting ensembl genes to {ensgc.output_directory}: vers │
│   22 │   │   )                                                               │
│   23 │   │   logging.info(f"connection_url: {ensgc.connection_url}")         │
│ ❱ 24 │   │   ensgc.export_datasets()                                         │
│   25 │                                                                       │
│   26 │   @staticmethod                                                       │
│   27 │   @cli.command(name="notebooks")                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │   ensgc = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer       │ │
│ │           object at 0x7f986b2355a0>                                      │ │
│ │ release = '111'                                                          │ │
│ │ species = 'human'                                                        │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :549 in export_datasets                                                      │
│                                                                              │
│   546 │   def export_datasets(self) -> None:                                 │
│   547 │   │   for export in self.exports:                                    │
│   548 │   │   │   logging.info(f"exporting {export.name} data")              │
│ ❱ 549 │   │   │   self.write_dataset(export)                                 │
│   550 │                                                                      │
│   551 │   def write_dataset(self, export: DatasetExport) -> None:            │
│   552 │   │   df = getattr(self, export.query_fxn)                           │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ export = DatasetExport(                                                  │ │
│ │          │   name='genes',                                               │ │
│ │          │   query_fxn='gene_df',                                        │ │
│ │          │   description='Primary table of ensembl genes with IDs,       │ │
│ │          symbols, and genomic location informati'+150,                   │ │
│ │          │   export_formats=[                                            │ │
│ │          │   │   <ExportFormat.parquet: 'parquet'>,                      │ │
│ │          │   │   <ExportFormat.tsv: 'tsv'>,                              │ │
│ │          │   │   <ExportFormat.excel: 'excel'>,                          │ │
│ │          │   │   <ExportFormat.json: 'json'>                             │ │
│ │          │   ]                                                           │ │
│ │          )                                                               │ │
│ │   self = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer object │ │
│ │          at 0x7f986b2355a0>                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :552 in write_dataset                                                        │
│                                                                              │
│   549 │   │   │   self.write_dataset(export)                                 │
│   550 │                                                                      │
│   551 │   def write_dataset(self, export: DatasetExport) -> None:            │
│ ❱ 552 │   │   df = getattr(self, export.query_fxn)                           │
│   553 │   │   assert isinstance(df, pd.DataFrame)                            │
│   554 │   │   gz_compression = {"method": "gzip", "mtime": 0}                │
│   555 │   │   if ExportFormat.parquet in export.export_formats:              │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ export = DatasetExport(                                                  │ │
│ │          │   name='genes',                                               │ │
│ │          │   query_fxn='gene_df',                                        │ │
│ │          │   description='Primary table of ensembl genes with IDs,       │ │
│ │          symbols, and genomic location informati'+150,                   │ │
│ │          │   export_formats=[                                            │ │
│ │          │   │   <ExportFormat.parquet: 'parquet'>,                      │ │
│ │          │   │   <ExportFormat.tsv: 'tsv'>,                              │ │
│ │          │   │   <ExportFormat.excel: 'excel'>,                          │ │
│ │          │   │   <ExportFormat.json: 'json'>                             │ │
│ │          │   ]                                                           │ │
│ │          )                                                               │ │
│ │   self = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer object │ │
│ │          at 0x7f986b2355a0>                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/functools.py:981 in   │
│ __get__                                                                      │
│                                                                              │
│   978 │   │   │   │   # check if another thread filled cache while we awaite │
│   979 │   │   │   │   val = cache.get(self.attrname, _NOT_FOUND)             │
│   980 │   │   │   │   if val is _NOT_FOUND:                                  │
│ ❱ 981 │   │   │   │   │   val = self.func(instance)                          │
│   982 │   │   │   │   │   try:                                               │
│   983 │   │   │   │   │   │   cache[self.attrname] = val                     │
│   984 │   │   │   │   │   except TypeError:                                  │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │    cache = {                                                             │ │
│ │            │   'species': Species(                                       │ │
│ │            │   │   name='homo_sapiens',                                  │ │
│ │            │   │   common_name='human',                                  │ │
│ │            │   │   assembly='38',                                        │ │
│ │            │   │   ensembl_gene_pattern='^ENSG[0-9]{11}$',               │ │
│ │            │   │   enable_mhc=True,                                      │ │
│ │            │   │   mhc_chromosome='6',                                   │ │
│ │            │   │   mhc_lower=28510120,                                   │ │
│ │            │   │   mhc_upper=33480577,                                   │ │
│ │            │   │   xmhc_lower=25726063,                                  │ │
│ │            │   │   xmhc_upper=33410226,                                  │ │
│ │            │   │   chromosomes=[                                         │ │
│ │            │   │   │   '1',                                              │ │
│ │            │   │   │   '2',                                              │ │
│ │            │   │   │   '3',                                              │ │
│ │            │   │   │   '4',                                              │ │
│ │            │   │   │   '5',                                              │ │
│ │            │   │   │   '6',                                              │ │
│ │            │   │   │   '7',                                              │ │
│ │            │   │   │   '8',                                              │ │
│ │            │   │   │   '9',                                              │ │
│ │            │   │   │   '10',                                             │ │
│ │            │   │   │   ... +15                                           │ │
│ │            │   │   ]                                                     │ │
│ │            │   ),                                                        │ │
│ │            │   'release': '111',                                         │ │
│ │            │   'database': 'homo_sapiens_core_111_38',                   │ │
│ │            │   'output_directory':                                       │ │
│ │            PosixPath('output/homo_sapiens_core_111_38'),                 │ │
│ │            │   '_xref_raw_df':         ensembl_gene_id   xref_source     │ │
│ │            ... xref_info_type xref_linkage_annotation                    │ │
│ │            0       ENSG00000000003  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            1       ENSG00000000003    EntrezGene  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            2       ENSG00000000003     GeneCards  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            3       ENSG00000000003          HGNC  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            4       ENSG00000000003      MIM_GENE  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            ...                 ...           ...  ...            ...     │ │
│ │            ...                                                           │ │
│ │            505962  ENSG00000293556  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505963  ENSG00000293557  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505964  ENSG00000293558  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505965  ENSG00000293559  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505966  ENSG00000293560  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │                                                                          │ │
│ │            [505967 rows x 7 columns],                                    │ │
│ │            │   'xref_lrg_df':         ensembl_gene_id lrg_gene_id        │ │
│ │            89      ENSG00000000971      LRG_47                           │ │
│ │            144     ENSG00000001084    LRG_1166                           │ │
│ │            264     ENSG00000001626     LRG_663                           │ │
│ │            349     ENSG00000001631     LRG_650                           │ │
│ │            438     ENSG00000002586    LRG_1023                           │ │
│ │            ...                 ...         ...                           │ │
│ │            455788  ENSG00000277027     LRG_163                           │ │
│ │            458738  ENSG00000277586     LRG_259                           │ │
│ │            467390  ENSG00000279220    LRG_1105                           │ │
│ │            475102  ENSG00000282608     LRG_424                           │ │
│ │            476974  ENSG00000283122    LRG_1035                           │ │
│ │                                                                          │ │
│ │            [1324 rows x 2 columns]                                       │ │
│ │            }                                                             │ │
│ │ instance = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer      │ │
│ │            object at 0x7f986b2355a0>                                     │ │
│ │    owner = <class                                                        │ │
│ │            'ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer'>    │ │
│ │     self = <functools.cached_property object at 0x7f9876531780>          │ │
│ │      val = <object object at 0x7f9884334340>                             │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :99 in gene_df                                                               │
│                                                                              │
│    96 │   │   gene_df = gene_df.join(desc_df)                                │
│    97 │   │   # add ensembl_representative_gene_id column                    │
│    98 │   │   gene_repr_df = gene_df.merge(                                  │
│ ❱  99 │   │   │   self.alt_allele_df[["ensembl_gene_id", "ensembl_representa │
│   100 │   │   │   how="left",                                                │
│   101 │   │   )                                                              │
│   102 │   │   gene_repr_df.ensembl_representative_gene_id = (                │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ desc_df = │   │   │   │   │   │   │   │   │   │   gene_description  ...  │ │
│ │           gene_description_source_id                                     │ │
│ │           0                                          tetraspanin 6  ...  │ │
│ │           HGNC:11858                                                     │ │
│ │           1                                            tenomodulin  ...  │ │
│ │           HGNC:17757                                                     │ │
│ │           2      dolichyl-phosphate mannosyltransferase subunit...  ...  │ │
│ │           HGNC:3005                                                      │ │
│ │           3                               SCY1 like pseudokinase 3  ...  │ │
│ │           HGNC:19285                                                     │ │
│ │           4      FIGNL1 interacting regulator of recombination ...  ...  │ │
│ │           HGNC:25565                                                     │ │
│ │           ...                                                  ...  ...  │ │
│ │           ...                                                            │ │
│ │           70706                                   novel transcript  ...  │ │
│ │           NaN                                                            │ │
│ │           70707                                   novel transcript  ...  │ │
│ │           NaN                                                            │ │
│ │           70708                                   novel transcript  ...  │ │
│ │           NaN                                                            │ │
│ │           70709                                   novel transcript  ...  │ │
│ │           NaN                                                            │ │
│ │           70710                                   novel transcript  ...  │ │
│ │           NaN                                                            │ │
│ │                                                                          │ │
│ │           [70711 rows x 3 columns]                                       │ │
│ │ gene_df = │      ensembl_gene_id  ...  gene_description_source_id        │ │
│ │           0      ENSG00000000003  ...                  HGNC:11858        │ │
│ │           1      ENSG00000000005  ...                  HGNC:17757        │ │
│ │           2      ENSG00000000419  ...                   HGNC:3005        │ │
│ │           3      ENSG00000000457  ...                  HGNC:19285        │ │
│ │           4      ENSG00000000460  ...                  HGNC:25565        │ │
│ │           ...                ...  ...                         ...        │ │
│ │           70706  ENSG00000293556  ...                         NaN        │ │
│ │           70707  ENSG00000293557  ...                         NaN        │ │
│ │           70708  ENSG00000293558  ...                         NaN        │ │
│ │           70709  ENSG00000293559  ...                         NaN        │ │
│ │           70710  ENSG00000293560  ...                         NaN        │ │
│ │                                                                          │ │
│ │           [70711 rows x 23 columns]                                      │ │
│ │    self = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer       │ │
│ │           object at 0x7f986b2355a0>                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/functools.py:981 in   │
│ __get__                                                                      │
│                                                                              │
│   978 │   │   │   │   # check if another thread filled cache while we awaite │
│   979 │   │   │   │   val = cache.get(self.attrname, _NOT_FOUND)             │
│   980 │   │   │   │   if val is _NOT_FOUND:                                  │
│ ❱ 981 │   │   │   │   │   val = self.func(instance)                          │
│   982 │   │   │   │   │   try:                                               │
│   983 │   │   │   │   │   │   cache[self.attrname] = val                     │
│   984 │   │   │   │   │   except TypeError:                                  │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │    cache = {                                                             │ │
│ │            │   'species': Species(                                       │ │
│ │            │   │   name='homo_sapiens',                                  │ │
│ │            │   │   common_name='human',                                  │ │
│ │            │   │   assembly='38',                                        │ │
│ │            │   │   ensembl_gene_pattern='^ENSG[0-9]{11}$',               │ │
│ │            │   │   enable_mhc=True,                                      │ │
│ │            │   │   mhc_chromosome='6',                                   │ │
│ │            │   │   mhc_lower=28510120,                                   │ │
│ │            │   │   mhc_upper=33480577,                                   │ │
│ │            │   │   xmhc_lower=25726063,                                  │ │
│ │            │   │   xmhc_upper=33410226,                                  │ │
│ │            │   │   chromosomes=[                                         │ │
│ │            │   │   │   '1',                                              │ │
│ │            │   │   │   '2',                                              │ │
│ │            │   │   │   '3',                                              │ │
│ │            │   │   │   '4',                                              │ │
│ │            │   │   │   '5',                                              │ │
│ │            │   │   │   '6',                                              │ │
│ │            │   │   │   '7',                                              │ │
│ │            │   │   │   '8',                                              │ │
│ │            │   │   │   '9',                                              │ │
│ │            │   │   │   '10',                                             │ │
│ │            │   │   │   ... +15                                           │ │
│ │            │   │   ]                                                     │ │
│ │            │   ),                                                        │ │
│ │            │   'release': '111',                                         │ │
│ │            │   'database': 'homo_sapiens_core_111_38',                   │ │
│ │            │   'output_directory':                                       │ │
│ │            PosixPath('output/homo_sapiens_core_111_38'),                 │ │
│ │            │   '_xref_raw_df':         ensembl_gene_id   xref_source     │ │
│ │            ... xref_info_type xref_linkage_annotation                    │ │
│ │            0       ENSG00000000003  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            1       ENSG00000000003    EntrezGene  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            2       ENSG00000000003     GeneCards  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            3       ENSG00000000003          HGNC  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            4       ENSG00000000003      MIM_GENE  ...      DEPENDENT     │ │
│ │            None                                                          │ │
│ │            ...                 ...           ...  ...            ...     │ │
│ │            ...                                                           │ │
│ │            505962  ENSG00000293556  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505963  ENSG00000293557  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505964  ENSG00000293558  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505965  ENSG00000293559  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │            505966  ENSG00000293560  ArrayExpress  ...         DIRECT     │ │
│ │            None                                                          │ │
│ │                                                                          │ │
│ │            [505967 rows x 7 columns],                                    │ │
│ │            │   'xref_lrg_df':         ensembl_gene_id lrg_gene_id        │ │
│ │            89      ENSG00000000971      LRG_47                           │ │
│ │            144     ENSG00000001084    LRG_1166                           │ │
│ │            264     ENSG00000001626     LRG_663                           │ │
│ │            349     ENSG00000001631     LRG_650                           │ │
│ │            438     ENSG00000002586    LRG_1023                           │ │
│ │            ...                 ...         ...                           │ │
│ │            455788  ENSG00000277027     LRG_163                           │ │
│ │            458738  ENSG00000277586     LRG_259                           │ │
│ │            467390  ENSG00000279220    LRG_1105                           │ │
│ │            475102  ENSG00000282608     LRG_424                           │ │
│ │            476974  ENSG00000283122    LRG_1035                           │ │
│ │                                                                          │ │
│ │            [1324 rows x 2 columns]                                       │ │
│ │            }                                                             │ │
│ │ instance = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer      │ │
│ │            object at 0x7f986b2355a0>                                     │ │
│ │    owner = <class                                                        │ │
│ │            'ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer'>    │ │
│ │     self = <functools.cached_property object at 0x7f986b234b20>          │ │
│ │      val = <object object at 0x7f9884334340>                             │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :142 in alt_allele_df                                                        │
│                                                                              │
│   139 │   │   │   "representative_gene_method",                              │
│   140 │   │   ]                                                              │
│   141 │   │   if not alt_allele_df.empty:                                    │
│ ❱ 142 │   │   │   alt_allele_df = alt_allele_df.groupby("alt_allele_group_id │
│   143 │   │   │   │   self._alt_allele_add_representative                    │
│   144 │   │   │   )                                                          │
│   145 │   │   else:                                                          │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ alt_allele_df = │      ensembl_gene_id  ...  ensembl_created_date        │ │
│ │                 0      ENSG00000282572  ...   2015-06-01 18:57:05        │ │
│ │                 1      ENSG00000281951  ...   2015-06-01 18:57:05        │ │
│ │                 2      ENSG00000282572  ...   2015-06-01 18:57:05        │ │
│ │                 3      ENSG00000273644  ...   2014-06-09 10:49:07        │ │
│ │                 4      ENSG00000273644  ...   2014-06-09 10:49:07        │ │
│ │                 ...                ...  ...                   ...        │ │
│ │                 14506  ENSG00000284613  ...   2017-06-13 10:44:55        │ │
│ │                 14507  ENSG00000292409  ...   2023-04-14 17:13:51        │ │
│ │                 14508  ENSG00000254581  ...   2010-11-01 15:31:55        │ │
│ │                 14509  ENSG00000254581  ...   2010-11-01 15:31:55        │ │
│ │                 14510  ENSG00000292410  ...   2023-04-14 17:13:51        │ │
│ │                                                                          │ │
│ │                 [14511 rows x 7 columns]                                 │ │
│ │ expected_cols = [                                                        │ │
│ │                 │   'ensembl_gene_id',                                   │ │
│ │                 │   'alt_allele_group_id',                               │ │
│ │                 │   'alt_allele_is_representative',                      │ │
│ │                 │   'primary_assembly',                                  │ │
│ │                 │   'seq_region',                                        │ │
│ │                 │   'alt_allele_attrib',                                 │ │
│ │                 │   'ensembl_created_date',                              │ │
│ │                 │   'ensembl_representative_gene_id',                    │ │
│ │                 │   'is_representative_gene',                            │ │
│ │                 │   'representative_gene_method'                         │ │
│ │                 ]                                                        │ │
│ │          self = <ensembl_genes.ensembl_genes.Ensembl_Gene_Catalog_Writer │ │
│ │                 object at 0x7f986b2355a0>                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/.cache/pypoetry/virtualenvs/ensembl-genes-GU6ps7Hy-py3.10/lib/p │
│ ython3.10/site-packages/pandas/core/groupby/groupby.py:1567 in apply         │
│                                                                              │
│   1564 │   │   │   │   with rewrite_warning(                                 │
│   1565 │   │   │   │   │   old_msg, FutureWarning, new_msg                   │
│   1566 │   │   │   │   ) if is_np_func else nullcontext():                   │
│ ❱ 1567 │   │   │   │   │   result = self._python_apply_general(f, self._sele │
│   1568 │   │   │   except TypeError:                                         │
│   1569 │   │   │   │   # gh-20949                                            │
│   1570 │   │   │   │   # try again, with .apply acting as a filtering        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │       args = ()                                                          │ │
│ │          f = <function                                                   │ │
│ │              Ensembl_Gene_Queries._alt_allele_add_representative at      │ │
│ │              0x7f986b26fb50>                                             │ │
│ │       func = <function                                                   │ │
│ │              Ensembl_Gene_Queries._alt_allele_add_representative at      │ │
│ │              0x7f986b26fb50>                                             │ │
│ │ is_np_func = False                                                       │ │
│ │     kwargs = {}                                                          │ │
│ │    new_msg = 'The operation <function                                    │ │
│ │              Ensembl_Gene_Queries._alt_allele_add_representative at      │ │
│ │              0'+160                                                      │ │
│ │    old_msg = 'The default value of numeric_only'                         │ │
│ │  orig_func = <function                                                   │ │
│ │              Ensembl_Gene_Queries._alt_allele_add_representative at      │ │
│ │              0x7f986b26fb50>                                             │ │
│ │       self = <pandas.core.groupby.generic.DataFrameGroupBy object at     │ │
│ │              0x7f986a48ff10>                                             │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/.cache/pypoetry/virtualenvs/ensembl-genes-GU6ps7Hy-py3.10/lib/p │
│ ython3.10/site-packages/pandas/core/groupby/groupby.py:1629 in               │
│ _python_apply_general                                                        │
│                                                                              │
│   1626 │   │   Series or DataFrame                                           │
│   1627 │   │   │   data after applying f                                     │
│   1628 │   │   """                                                           │
│ ❱ 1629 │   │   values, mutated = self.grouper.apply(f, data, self.axis)      │
│   1630 │   │   if not_indexed_same is None:                                  │
│   1631 │   │   │   not_indexed_same = mutated or self.mutated                │
│   1632 │   │   override_group_keys = False                                   │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │             data = │      ensembl_gene_id  ...  ensembl_created_date     │ │
│ │                    0      ENSG00000282572  ...   2015-06-01 18:57:05     │ │
│ │                    1      ENSG00000281951  ...   2015-06-01 18:57:05     │ │
│ │                    2      ENSG00000282572  ...   2015-06-01 18:57:05     │ │
│ │                    3      ENSG00000273644  ...   2014-06-09 10:49:07     │ │
│ │                    4      ENSG00000273644  ...   2014-06-09 10:49:07     │ │
│ │                    ...                ...  ...                   ...     │ │
│ │                    14506  ENSG00000284613  ...   2017-06-13 10:44:55     │ │
│ │                    14507  ENSG00000292409  ...   2023-04-14 17:13:51     │ │
│ │                    14508  ENSG00000254581  ...   2010-11-01 15:31:55     │ │
│ │                    14509  ENSG00000254581  ...   2010-11-01 15:31:55     │ │
│ │                    14510  ENSG00000292410  ...   2023-04-14 17:13:51     │ │
│ │                                                                          │ │
│ │                    [14511 rows x 7 columns]                              │ │
│ │                f = <function                                             │ │
│ │                    Ensembl_Gene_Queries._alt_allele_add_representative   │ │
│ │                    at 0x7f986b26fb50>                                    │ │
│ │           is_agg = False                                                 │ │
│ │     is_transform = False                                                 │ │
│ │ not_indexed_same = None                                                  │ │
│ │             self = <pandas.core.groupby.generic.DataFrameGroupBy object  │ │
│ │                    at 0x7f986a48ff10>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/.cache/pypoetry/virtualenvs/ensembl-genes-GU6ps7Hy-py3.10/lib/p │
│ ython3.10/site-packages/pandas/core/groupby/ops.py:839 in apply              │
│                                                                              │
│    836 │   │   │                                                             │
│    837 │   │   │   # group might be modified                                 │
│    838 │   │   │   group_axes = group.axes                                   │
│ ❱  839 │   │   │   res = f(group)                                            │
│    840 │   │   │   if not mutated and not _is_indexed_like(res, group_axes,  │
│    841 │   │   │   │   mutated = True                                        │
│    842 │   │   │   result_values.append(res)                                 │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          axis = 0                                                        │ │
│ │          data = │      ensembl_gene_id  ...  ensembl_created_date        │ │
│ │                 0      ENSG00000282572  ...   2015-06-01 18:57:05        │ │
│ │                 1      ENSG00000281951  ...   2015-06-01 18:57:05        │ │
│ │                 2      ENSG00000282572  ...   2015-06-01 18:57:05        │ │
│ │                 3      ENSG00000273644  ...   2014-06-09 10:49:07        │ │
│ │                 4      ENSG00000273644  ...   2014-06-09 10:49:07        │ │
│ │                 ...                ...  ...                   ...        │ │
│ │                 14506  ENSG00000284613  ...   2017-06-13 10:44:55        │ │
│ │                 14507  ENSG00000292409  ...   2023-04-14 17:13:51        │ │
│ │                 14508  ENSG00000254581  ...   2010-11-01 15:31:55        │ │
│ │                 14509  ENSG00000254581  ...   2010-11-01 15:31:55        │ │
│ │                 14510  ENSG00000292410  ...   2023-04-14 17:13:51        │ │
│ │                                                                          │ │
│ │                 [14511 rows x 7 columns]                                 │ │
│ │             f = <function                                                │ │
│ │                 Ensembl_Gene_Queries._alt_allele_add_representative at   │ │
│ │                 0x7f986b26fb50>                                          │ │
│ │         group = │    ensembl_gene_id  ...  ensembl_created_date          │ │
│ │                 501  ENSG00000273592  ...   2014-06-09 10:49:07          │ │
│ │                 502  ENSG00000276935  ...   2014-06-09 10:49:07          │ │
│ │                                                                          │ │
│ │                 [2 rows x 7 columns]                                     │ │
│ │    group_axes = [                                                        │ │
│ │                 │   Int64Index([501, 502], dtype='int64'),               │ │
│ │                 │   Index(['ensembl_gene_id', 'alt_allele_group_id',     │ │
│ │                 │      'alt_allele_is_representative',                   │ │
│ │                 'primary_assembly', 'seq_region',                        │ │
│ │                 │      'alt_allele_attrib', 'ensembl_created_date'],     │ │
│ │                 │     dtype='object')                                    │ │
│ │                 ]                                                        │ │
│ │    group_keys = Int64Index([44429, 44430, 44431, 44432, 44433, 44434,    │ │
│ │                 44435, 44436, 44437,                                     │ │
│ │                 │   │   │   44438,                                       │ │
│ │                 │   │   │   ...                                          │ │
│ │                 │   │   │   48534, 48535, 48536, 48537, 48538, 48539,    │ │
│ │                 48540, 48541, 48542,                                     │ │
│ │                 │   │   │   48543],                                      │ │
│ │                 │   │      dtype='int64', name='alt_allele_group_id',    │ │
│ │                 length=3993)                                             │ │
│ │           key = 44458                                                    │ │
│ │       mutated = False                                                    │ │
│ │           res = │    ensembl_gene_id  ...    representative_gene_method  │ │
│ │                 497  ENSG00000094796  ...  alt_allele_is_representative  │ │
│ │                 498  ENSG00000094796  ...  alt_allele_is_representative  │ │
│ │                 499  ENSG00000262993  ...  alt_allele_is_representative  │ │
│ │                 500  ENSG00000292029  ...  alt_allele_is_representative  │ │
│ │                                                                          │ │
│ │                 [4 rows x 10 columns]                                    │ │
│ │ result_values = [                                                        │ │
│ │                 │      ensembl_gene_id  ...                              │ │
│ │                 representative_gene_method                               │ │
│ │                 0  ENSG00000282572  ...  alt_allele_is_representative    │ │
│ │                 1  ENSG00000281951  ...  alt_allele_is_representative    │ │
│ │                 2  ENSG00000282572  ...  alt_allele_is_representative    │ │
│ │                                                                          │ │
│ │                 [3 rows x 10 columns],                                   │ │
│ │                 │      ensembl_gene_id  ...                              │ │
│ │                 representative_gene_method                               │ │
│ │                 3  ENSG00000273644  ...  alt_allele_is_representative    │ │
│ │                 4  ENSG00000273644  ...  alt_allele_is_representative    │ │
│ │                 5  ENSG00000282333  ...  alt_allele_is_representative    │ │
│ │                                                                          │ │
│ │                 [3 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 6   ENSG00000232325  ...  alt_allele_is_representative   │ │
│ │                 7   ENSG00000232325  ...  alt_allele_is_representative   │ │
│ │                 8   ENSG00000281993  ...  alt_allele_is_representative   │ │
│ │                 9   ENSG00000282645  ...  alt_allele_is_representative   │ │
│ │                 10  ENSG00000288372  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 11  ENSG00000242611  ...  alt_allele_is_representative   │ │
│ │                 12  ENSG00000242611  ...  alt_allele_is_representative   │ │
│ │                 13  ENSG00000282155  ...  alt_allele_is_representative   │ │
│ │                 14  ENSG00000282557  ...  alt_allele_is_representative   │ │
│ │                 15  ENSG00000288288  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 16  ENSG00000242474  ...  alt_allele_is_representative   │ │
│ │                 17  ENSG00000242474  ...  alt_allele_is_representative   │ │
│ │                 18  ENSG00000282226  ...  alt_allele_is_representative   │ │
│ │                 19  ENSG00000282662  ...  alt_allele_is_representative   │ │
│ │                 20  ENSG00000288472  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 21  ENSG00000240859  ...  alt_allele_is_representative   │ │
│ │                 22  ENSG00000240859  ...  alt_allele_is_representative   │ │
│ │                 23  ENSG00000282075  ...  alt_allele_is_representative   │ │
│ │                 24  ENSG00000282461  ...  alt_allele_is_representative   │ │
│ │                 25  ENSG00000288417  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 26  ENSG00000261795  ...  alt_allele_is_representative   │ │
│ │                 27  ENSG00000261795  ...  alt_allele_is_representative   │ │
│ │                 28  ENSG00000281767  ...  alt_allele_is_representative   │ │
│ │                 29  ENSG00000282781  ...  alt_allele_is_representative   │ │
│ │                 30  ENSG00000288502  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 31  ENSG00000239715  ...  alt_allele_is_representative   │ │
│ │                 32  ENSG00000239715  ...  alt_allele_is_representative   │ │
│ │                 33  ENSG00000281349  ...  alt_allele_is_representative   │ │
│ │                 34  ENSG00000282309  ...  alt_allele_is_representative   │ │
│ │                 35  ENSG00000288311  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 36  ENSG00000240093  ...  alt_allele_is_representative   │ │
│ │                 37  ENSG00000240093  ...  alt_allele_is_representative   │ │
│ │                 38  ENSG00000281788  ...  alt_allele_is_representative   │ │
│ │                 39  ENSG00000282575  ...  alt_allele_is_representative   │ │
│ │                 40  ENSG00000288449  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   │   ensembl_gene_id  ...                             │ │
│ │                 representative_gene_method                               │ │
│ │                 41  ENSG00000177706  ...  alt_allele_is_representative   │ │
│ │                 42  ENSG00000177706  ...  alt_allele_is_representative   │ │
│ │                 43  ENSG00000281429  ...  alt_allele_is_representative   │ │
│ │                 44  ENSG00000282147  ...  alt_allele_is_representative   │ │
│ │                 45  ENSG00000288499  ...  alt_allele_is_representative   │ │
│ │                                                                          │ │
│ │                 [5 rows x 10 columns],                                   │ │
│ │                 │   ... +19                                              │ │
│ │                 ]                                                        │ │
│ │          self = <pandas.core.groupby.ops.BaseGrouper object at           │ │
│ │                 0x7f986a48fd60>                                          │ │
│ │      splitter = <pandas.core.groupby.ops.FrameSplitter object at         │ │
│ │                 0x7f986a48e080>                                          │ │
│ │        zipped = <zip object at 0x7f9856a41e80>                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :203 in _alt_allele_add_representative                                       │
│                                                                              │
│   200 │   │   Apply to alt_allele_df grouped by `alt_allele_group_id` to add │
│   201 │   │   `ensembl_representative_gene_id`, `is_representative_gene`, `r │
│   202 │   │   """                                                            │
│ ❱ 203 │   │   representative, method = Ensembl_Gene_Queries._alt_allele_get_ │
│   204 │   │   df["ensembl_representative_gene_id"] = representative          │
│   205 │   │   df["is_representative_gene"] = (                               │
│   206 │   │   │   df.ensembl_gene_id == df.ensembl_representative_gene_id    │
│                                                                              │
│ ╭─────────────────────── locals ───────────────────────╮                     │
│ │ df = │    ensembl_gene_id  ...  ensembl_created_date │                     │
│ │      501  ENSG00000273592  ...   2014-06-09 10:49:07 │                     │
│ │      502  ENSG00000276935  ...   2014-06-09 10:49:07 │                     │
│ │                                                      │                     │
│ │      [2 rows x 7 columns]                            │                     │
│ ╰──────────────────────────────────────────────────────╯                     │
│                                                                              │
│ /home/runner/work/ensembl-genes/ensembl-genes/ensembl_genes/ensembl_genes.py │
│ :187 in _alt_allele_get_representative                                       │
│                                                                              │
│   184 │   │   if len(representatives) == 1:                                  │
│   185 │   │   │   return representatives[0], "primary_assembly"              │
│   186 │   │   if len(representatives) > 1:                                   │
│ ❱ 187 │   │   │   raise ValueError(                                          │
│   188 │   │   │   │   "expected at most 1 primary assembly gene per alt_alle │
│   189 │   │   │   )                                                          │
│   190 │   │   return (                                                       │
│                                                                              │
│ ╭───────────────────────────── locals ──────────────────────────────╮        │
│ │              df = │    ensembl_gene_id  ...  ensembl_created_date │        │
│ │                   501  ENSG00000273592  ...   2014-06-09 10:49:07 │        │
│ │                   502  ENSG00000276935  ...   2014-06-09 10:49:07 │        │
│ │                                                                   │        │
│ │                   [2 rows x 7 columns]                            │        │
│ │ representatives = ['ENSG00000273592', 'ENSG00000276935']          │        │
│ ╰───────────────────────────────────────────────────────────────────╯        │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: expected at most 1 primary assembly gene per alt_allele_group
Error: Process completed with exit code 1.
@dhimmel
Copy link
Member

dhimmel commented Feb 5, 2024

I think #22 is the underlying issue here. Release 110 had an unsolvable issue:

It is not possible to link an alternate region with the corresponding primary assembly region in the release 110 database. This information will be present in the dna_align_feature table in the release 111 database.

But now that 111 is released, I'll look to update the SQL queries and logic for the new database schema. Thanks for the reminder!

@ACastanza
Copy link
Contributor Author

Any idea when you might be able to get this fix in? We make use of this history file and I'd like to use the current version instead of carrying forward the old one again.

@dhimmel
Copy link
Member

dhimmel commented Feb 14, 2024 •
edited
Loading

Noting 111 release information:

Any idea when you might be able to get this fix in?

Let me do a little work now to assess the feasibility.

dhimmel added a commit that referenced this issue Feb 14, 2024
@dhimmel
remove sql usage of assembly_exception table
9cdc34e 
refs #25
refs #22

As of ensembl 110, this table was emptied causing our detection of
primary assembly to be flawed. This adopts a slightly flawed approach
for detecting the primary assembly, but one that should unblock
builds. Chromosomes detection is also now incomplete.
dhimmel added a commit that referenced this issue Feb 15, 2024
@dhimmel
remove sql usage of assembly_exception table
eb7c779 
refs #25
refs #22

As of ensembl 110, this table was emptied causing our detection of
primary assembly to be flawed. This adopts a slightly flawed approach
for detecting the primary assembly, but one that should unblock
builds. Chromosomes detection is also now incomplete.
@dhimmel
Copy link
Member

dhimmel commented Feb 15, 2024

Unblocked by eb7c779 and rerunning exports in this action.

@dhimmel
Copy link
Member

dhimmel commented Feb 15, 2024

Okay 111 is working with the latest human output in 8c4decd.

The two source code commits that change methodology are:

We've switched to a more aggressive method for detection allele groups. There will no longer be any representative genes with duplicate symbols.

@ACastanza will let you close the issue if the new data looks good for you and your application.

@ACastanza
Copy link
Contributor Author

Looks good to me, thanks for getting this fix in, much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dhimmel @ACastanza