Wrong inferred return type for functions with overloads

[sk-]

In our codebase, which uses sqlalchemy, we have a bunch of errors due to how overloads are handled in presence of Any.

The following code shows the issue

from typing import Any

import sqlalchemy as sa
from sqlalchemy import Table
from sqlalchemy.sql.functions import sum


def build_label(table: Table, label: str) -> sa.Label[Any]:
    reveal_type(sa.func.sum(table.c.value_col))
    reveal_type(sa.func.coalesce(sa.func.sum(table.c.value_col), 0))
    return sa.func.coalesce(sa.func.sum(table.c.value_col), 0).label(label)


def build_label_v2(table: Table, label: str) -> sa.Label[Any]:
    sum_col: sum[Any] = sa.func.sum(table.c.value_col)
    return sa.func.coalesce(sum_col, 0).label(label)


def build_label_v3(table: Table, label: str) -> sa.Label[Any]:
    sum_col: sum[int] = sa.func.sum(table.c.value_col)
    return sa.func.coalesce(sum_col, 0).label(label)

In mypy, this type checks correctly and outputs:

zuban_sa_func.py:9: note: Revealed type is "sqlalchemy.sql.functions.sum[Any]"
zuban_sa_func.py:10: note: Revealed type is "sqlalchemy.sql.functions.coalesce[Any]"
Success: no issues found in 1 source file

while zuban reports

zuban_sa_func.py:9: note: Revealed type is "Any"
zuban_sa_func.py:10: note: Revealed type is "Any"
zuban_sa_func.py:11: error: Returning Any from function declared to return "Label[Any]"  [no-any-return]
zuban_sa_func.py:16: error: Returning Any from function declared to return "Label[Any]"  [no-any-return]
Found 2 errors in 1 file (checked 1 source file)

Note that we were able to correctly type the function build_label_v3 by completely avoiding the Any type.

I'm under the impression this may be related to #263.

[davidhalter]

This is a bad type annotation if you go to the definition as far as I can tell.

Function "sqlalchemy.sql.functions._FunctionGenerator.sum" is not valid as a type                                                                       
Perhaps you need "Callable[...]" or a callback protocol?                    

This is wrong according to the conformance tests. Here I'm changing the spec: python/typing#2144. (It's almost accepted and will be the future). I might do it if there is commercial support, but otherwise it's just not worth it. I'm closing for now.

[sk-]

@davidhalter I think you are confusing sqlalchemy.sql.functions._FunctionGenerator.sum (dynamic dispatch) with sqlalchemy.sql.functions.sum, which is properly defined here https://github.com/sqlalchemy/sqlalchemy/blob/94ed75eec98246c0c8a5cda8a82a779bd8b4f1c8/lib/sqlalchemy/sql/functions.py#L1805-L1808

Note that in the code above we are using the type sqlalchemy.sql.functions.sum for the sum_colvariable and that works without any warnings, in fact if I add reveal_type to the last 2 functions I get

zuban_sa_func.py:16: note: Revealed type is "sqlalchemy.sql.functions.sum[Any]"
zuban_sa_func.py:22: note: Revealed type is "sqlalchemy.sql.functions.sum[builtins.int]"

[davidhalter]

I'm not sure I understand what you are saying. But I'm sure I just wasn't precise enough.

I'm talking about the annotationin sqlalchemy/sql/functions.py here:

        @overload
        def sum(  # noqa: A001
            self,
            col: _ColumnExpressionArgument[_T],            
            *args: _ColumnExpressionOrLiteralArgument[Any],
            **kwargs: Any,
        ) -> sum[_T]: ...

Here def sum(...) -> sum[...] which is circular according to the conformance tests and therefore a problem with sqlalchemy definitions. This essentially results in def sum(...) -> Any: .... You can also see this error when you goto the sum function.

[sk-]

Thanks for the clarification. However, that raises another question, why is then the 3rd function build_label_v3 valid? I would also expect coalesce to be considered to return Any as their types are also recursive according to your explanation.

Coalesce is defined as

        @overload
        def coalesce(
            self,
            col: ColumnElement[_T],
            *args: _ColumnExpressionOrLiteralArgument[Any],
            **kwargs: Any,
        ) -> coalesce[_T]: ...

And if I check the following function

def build_label_v3(table: Table, label: str) -> sa.Label[Any]:
    sum_col: sum[int] = sa.func.sum(table.c.value_col)
    reveal_type(sa.func.coalesce(sum_col, 0))
    return sa.func.coalesce(sum_col, 0).label(label)

the revealed type of the coalesce expression is

zuban_sa_func.py:21: note: Revealed type is "sqlalchemy.sql.functions.coalesce[builtins.int]"

Also note that build_label_v2 does not type check, and I think it is related to how Anys are handled.

[davidhalter]

I think this is just a small bug that makes this confusing. I'll probably wait with fixing it until the conformance test change goes through. If you look at the errors, you can see that the first overload does not fail, but the other ones do. This is just how name binding currently works in Zuban, but that's probably wrong according to how PEP 749 works, which evaluates annotations now. The only reason why these type annotations work for sqlalchemy with 3.14 is that they have a from __future__ import annotations, which is now deprecated.

[sk-]

Thanks for the patience and you clarifications. I changed the type names to Sum and Coalesce and that makes it work.

Do you have any pointers to the conformance test where this is covered, so I can file a bug to Sqlalchemy?

[davidhalter]

I think the PEP I linked is the best reason. It's the reason why we are changing/streamlining conformance tests at python/typing#2144. Make sure to link the sqlalchemy issue here, I'm happy to chime in if there are questions.

The PEP itself evaluates annotations in a lazy way in the same scope where they are defined at. Mypy/Pyright already have different behaviors about that (even before the PEP) and they might not work properly even now, since they currently don't have the same behavior around this.

[sk-]

This is the issue I created sqlalchemy/sqlalchemy#13167