Skip to content

fix(postgres): treat FULLTEXT and SPATIAL as unreserved identifiers#2319

Open
fmguerreiro wants to merge 64 commits intoapache:mainfrom
fmguerreiro:fix/postgres-unreserve-fulltext
Open

fix(postgres): treat FULLTEXT and SPATIAL as unreserved identifiers#2319
fmguerreiro wants to merge 64 commits intoapache:mainfrom
fmguerreiro:fix/postgres-unreserve-fulltext

Conversation

@fmguerreiro
Copy link
Copy Markdown
Contributor

Problem

PostgreSQL schemas that use fulltext as a column name (common with tsvector/trigram search) failed to parse:

CREATE TABLE public.film (
    fulltext tsvector NOT NULL
);

Error: SQL parse error: sql parser error: Expected: a data type name, found: 'fulltext'

Root Cause

parse_optional_table_constraint consumed the FULLTEXT token before realising it was not a valid constraint keyword for the current dialect, then backtracked with prev_token(). The consume-then-backtrack pattern left the parser in a subtly broken state for non-MySQL/Generic dialects, causing the next parse_column_def_inner call to see the wrong current token.

Fix

Add a peek-based early-return guard in parse_optional_table_constraint before any token is consumed. For dialects other than MySQL/Generic, FULLTEXT and SPATIAL are valid identifiers and must not be touched by the constraint parser:

if name.is_none()
    && self
        .peek_one_of_keywords(&[Keyword::FULLTEXT, Keyword::SPATIAL])
        .is_some()
    && !dialect_of!(self is GenericDialect | MySqlDialect)
{
    return Ok(None);
}

Test

Added parse_fulltext_column_and_index_in_postgres in tests/sqlparser_postgres.rs covering:

  • CREATE TABLE film (fulltext TSVECTOR NOT NULL) — column named fulltext with tsvector type
  • CREATE INDEX film_fulltext_idx ON film USING gist (fulltext) — GiST index on a fulltext column

Version

Bumped to 0.60.13.

…R TABLE operations (#1)

PostgreSQL supports FORCE ROW LEVEL SECURITY and NO FORCE ROW LEVEL SECURITY
as ALTER TABLE operations. Add parsing support for both variants.
feat: parse EXCLUDE constraints for PostgreSQL
…m-main

# Conflicts:
#	Cargo.toml
#	src/ast/ddl.rs
#	src/ast/helpers/stmt_create_table.rs
#	src/ast/mod.rs
#	src/ast/spans.rs
#	src/ast/table_constraints.rs
#	src/parser/mod.rs
#	tests/sqlparser_postgres.rs
chore: merge upstream apache/datafusion-sqlparser-rs main
- Add HANDLER and VALIDATOR keywords to the keyword list
- Add FdwRoutineClause enum for HANDLER/NO HANDLER and VALIDATOR/NO VALIDATOR clauses
- Add CreateForeignDataWrapper struct for CREATE FOREIGN DATA WRAPPER
- Add CreateForeignTable struct for CREATE FOREIGN TABLE
- Export new types from ast::mod and add Statement variants
- Add spans.rs coverage returning Span::empty() for the new variants
- Dispatch on FOREIGN keyword in parse_create, branching on DATA WRAPPER or TABLE
- parse_create_foreign_data_wrapper: parses optional HANDLER/NO HANDLER,
  VALIDATOR/NO VALIDATOR, and OPTIONS clauses
- parse_create_foreign_table: parses IF NOT EXISTS, column list via parse_columns,
  required SERVER name, and optional OPTIONS clause
Round-trip tests via pg().verified_stmt for:
- FDW: name-only, HANDLER, NO HANDLER, NO VALIDATOR, combined HANDLER+VALIDATOR+OPTIONS
- FOREIGN TABLE: basic columns+SERVER, IF NOT EXISTS, table-level OPTIONS
Add Statement::CreateAggregate, CreateAggregate struct, CreateAggregateOption
enum, and AggregateModifyKind enum to represent PostgreSQL CREATE AGGREGATE
DDL. Options are stored as a typed enum covering all documented parameters
(SFUNC, STYPE, FINALFUNC, PARALLEL, moving-aggregate variants, etc.).
Wire AGGREGATE into the CREATE dispatch (before the or_replace error branch
so CREATE OR REPLACE AGGREGATE is accepted). parse_create_aggregate parses
the name, argument-type list, and the options block. Each recognised option
keyword dispatches to parse_create_aggregate_option which produces the typed
CreateAggregateOption variant.
Three tests covering: basic old-style aggregate (SFUNC/STYPE/FINALFUNC/INITCOND),
CREATE OR REPLACE with PARALLEL = SAFE, and moving-aggregate options
(MSFUNC/MINVFUNC/MSTYPE/MFINALFUNC_EXTRA/MFINALFUNC_MODIFY). All use
pg().verified_stmt() to assert parse-then-display round-trips identically.
feat: parse CREATE TEXT SEARCH CONFIGURATION/DICTIONARY/PARSER/TEMPLATE
# Conflicts:
#	src/ast/ddl.rs
#	src/ast/mod.rs
# Conflicts:
#	src/ast/ddl.rs
#	src/ast/mod.rs
#	src/parser/mod.rs
# Conflicts:
#	src/ast/ddl.rs
#	src/ast/mod.rs
#	tests/sqlparser_postgres.rs
fix(spans): add Span arm for CreateAggregate
Add PublicationTarget enum (AllTables, Tables, TablesInSchema),
CreatePublication struct, and CreateSubscription struct to ddl.rs.
Add Statement variants, Display arms, and re-exports in mod.rs.
Add Span::empty() arms in spans.rs.
Add PUBLICATION and SUBSCRIPTION to keywords.rs.
…rFunctionKind with Procedure, add SetTablespace to AlterTableOperation and AlterIndexOperation

Adds AST types and Display impls for:
- AlterDomain + AlterDomainOperation (ADD/DROP CONSTRAINT, RENAME, OWNER TO, SET SCHEMA/DEFAULT, DROP DEFAULT, VALIDATE CONSTRAINT)
- AlterTrigger + AlterTriggerOperation (RENAME TO)
- AlterExtension + AlterExtensionOperation (UPDATE TO, SET SCHEMA, OWNER TO, RENAME TO)
- AlterFunctionKind::Procedure variant for ALTER PROCEDURE
- AlterTableOperation::SetTablespace
- AlterIndexOperation::SetTablespace

Updates Spanned impls to remain exhaustive.
feat: parse CREATE PUBLICATION and CREATE SUBSCRIPTION
# Conflicts:
#	src/ast/ddl.rs
#	tests/sqlparser_postgres.rs
feat: parse ALTER TABLE/INDEX SET TABLESPACE, ALTER DOMAIN, ALTER TRIGGER, ALTER EXTENSION, ALTER PROCEDURE
… nodes

Add SecurityLabelObjectKind enum (TABLE/COLUMN/DATABASE/DOMAIN/FUNCTION/
ROLE/SCHEMA/SEQUENCE/TYPE/VIEW/MATERIALIZED VIEW) and SecurityLabel struct,
CreateUserMapping struct with UserMappingUser enum, and CreateTablespace struct
to ddl.rs with standard derives, Display impls, and From<X> for Statement.
Add Statement variants, Display arms, re-exports in mod.rs.
Add Span::empty() arms in spans.rs.
Add LABEL and MAPPING to keywords.rs.

Refs: pgmold-115, pgmold-119, pgmold-120
…SPACE

Add parse_security_label to parse_statement dispatch on Keyword::SECURITY.
Add parse_create_user_mapping dispatched from CREATE USER MAPPING branch
(checked before existing CREATE USER fallthrough).
Add parse_create_tablespace dispatched from parse_create on Keyword::TABLESPACE.

Refs: pgmold-115, pgmold-119, pgmold-120
…REATE TABLESPACE

Add 4 tests for SECURITY LABEL covering TABLE, NULL label, ROLE, and SCHEMA targets.
Add 4 tests for CREATE USER MAPPING covering basic, IF NOT EXISTS with OPTIONS,
CURRENT_USER, and PUBLIC user specifiers.
Add 3 tests for CREATE TABLESPACE covering basic, OWNER, and WITH options.

Refs: pgmold-115, pgmold-119, pgmold-120
…er, CreateTransform AST nodes

Add structs and enums for four PostgreSQL-specific DDL statements:
- CreateStatistics with StatisticsKind (NDistinct, Dependencies, Mcv)
- CreateAccessMethod with AccessMethodType (Index, Table)
- CreateEventTrigger with EventTriggerEvent (DdlCommandStart, DdlCommandEnd, TableRewrite, SqlDrop)
- CreateTransform / TransformElement with OR REPLACE support

Adds TRANSFORM keyword to keywords.rs.

Closes pgmold-103, pgmold-104, pgmold-105, pgmold-106.
…TRANSFORM

Add parse_create_statistics, parse_create_access_method,
parse_create_event_trigger, and parse_create_transform to the parser.

- STATISTICS: name, optional IF NOT EXISTS, optional (kind, ...) list, ON expr-list, FROM table
- ACCESS METHOD: name TYPE INDEX|TABLE HANDLER function
- EVENT TRIGGER: name ON event [WHEN TAG IN (...)] EXECUTE FUNCTION|PROCEDURE name()
- TRANSFORM: [OR REPLACE] FOR type LANGUAGE lang (element, ...)

TRANSFORM is dispatched before the or_replace error guard.
…le AST nodes

Add AST structs and Display impls for four PostgreSQL catalog DDL
statements. Each follows the established fork pattern: struct in ddl.rs
with Debug/Clone/PartialEq/etc derives, Statement enum variant, Display
arm, From impl, and Span::empty() arm in spans.rs.

New keywords: ALSO, ASSIGNMENT, CONVERSION, IMPLICIT, INLINE,
PROCEDURAL, TRUSTED.

Closes pgmold-89, pgmold-91, pgmold-92, pgmold-93.
feat: parse SECURITY LABEL, CREATE USER MAPPING, CREATE TABLESPACE
# Conflicts:
#	src/ast/ddl.rs
#	src/ast/mod.rs
#	src/ast/spans.rs
#	src/parser/mod.rs
#	tests/sqlparser_postgres.rs
feat: parse CREATE STATISTICS, CREATE ACCESS METHOD, CREATE EVENT TRIGGER, CREATE TRANSFORM
# Conflicts:
#	src/ast/ddl.rs
#	src/ast/mod.rs
#	src/ast/spans.rs
#	src/parser/mod.rs
#	tests/sqlparser_postgres.rs
feat: parse CREATE CAST, CREATE CONVERSION, CREATE LANGUAGE, CREATE RULE
PRs #14, #15, and #16 each added new AST types that collide in the
same places in src/ast/ddl.rs, src/ast/mod.rs, src/parser/mod.rs,
and tests/sqlparser_postgres.rs. Several mid-match-arm and
mid-function-body concat resolutions during the three rebase-and-merge
passes left fn / impl / match blocks missing their closing braces
and left the ddl re-export list with duplicate entries.

This restores compile:
- Adds missing closing braces for CastFunctionKind, StatisticsKind,
  CreateRule / CreateStatistics / CreateEventTrigger / CreateTransform /
  CreateLanguage Display impls and parse fns, plus the CreateTransform
  From-to-Statement impl.
- Dedupes and alphabetizes the `pub use self::ddl::{ ... }` list in
  src/ast/mod.rs.
- Adds missing doc comments on CastFunctionKind fields and RuleEvent
  variants that #[warn(missing_docs)] requires.

Unblocks 0.60.10 publish.
fix: close delimiter gaps left by concat merges
PostgreSQL's pg_dump emits WITH NO DATA on unpopulated materialized views.
Add a with_data: Option<bool> field to CreateView and parse the clause
after the query body. Bump to 0.60.11.
This was an accidentally-committed gitlink (160000 mode) with no
corresponding entry in .gitmodules. It does not break git itself
but causes any cargo git-dep consumer to fail cloning the repo:

  failed to update submodule `.worktrees/fix-pr-2307-ci`
  no URL configured for submodule '.worktrees/fix-pr-2307-ci'

Same class of residue that 837b5a0 on feat/exclude-constraint-upstream
cleaned up, but that commit never reached main so the gitlink still
ships on every descendant branch.
feat(parser): parse WITH [NO] DATA on CREATE MATERIALIZED VIEW
Add AlterTableOperation::AttachPartitionOf and DetachPartitionOf variants
for PostgreSQL declarative partition DDL:

  ALTER TABLE parent ATTACH PARTITION child { FOR VALUES <spec> | DEFAULT }
  ALTER TABLE parent DETACH PARTITION child [ CONCURRENTLY | FINALIZE ]

These are distinct from the existing ClickHouse AttachPartition/DetachPartition
variants which use the PART|PARTITION <expr> syntax.

- Add FINALIZE keyword to the keyword list
- Add span impls for the new variants
- Add parser tests covering range, list, hash, default attach and
  plain, concurrently, finalize detach
- Bump version to 0.60.12
…h-partition

feat: parse ALTER TABLE ATTACH/DETACH PARTITION for PostgreSQL
FULLTEXT and SPATIAL are MySQL-specific table constraint keywords. When
parse_optional_table_constraint encountered them in a PostgreSQL context
it would consume the token, fall through to the _ arm, and call
prev_token() to restore it — but the consume-then-backtrack round-trip
left room for a subtle parser-state bug that caused the next call to
parse_data_type to see 'fulltext' instead of the actual column type.

Fix: add an explicit peek-based guard at the top of
parse_optional_table_constraint so that for dialects other than MySQL /
Generic the FULLTEXT / SPATIAL tokens are never consumed at all. The
caller (parse_columns) then correctly falls through to parse_column_def
and treats the word as an ordinary column identifier.

Also add a parser unit test covering a tsvector column literally named
'fulltext' (as in the pagila DVD-rental schema) and a GiST index that
references the same column by name.

Bump version to 0.60.13.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant