Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-79579: Improve DML query detection in sqlite3 #93623

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

erlend-aasland
Copy link
Contributor

@erlend-aasland erlend-aasland commented Jun 8, 2022

Strip whitespace and comments from queries in order to harden DML query
detection.

Resolves #79579

@erlend-aasland
Copy link
Contributor Author

@erlend-aasland erlend-aasland commented Jun 8, 2022

@erlend-aasland
Copy link
Contributor Author

@erlend-aasland erlend-aasland commented Jun 8, 2022

If the sqlite3_normalized_sql API becomes enabled by default in the future, we can get rid of our own parser helper, and just use sqlite3_normalized_sql to strip whitespace and comments. We'll see what the future brings.

Modules/_sqlite/statement.c Outdated Show resolved Hide resolved
@erlend-aasland erlend-aasland added the 🔨 test-with-buildbots label Jun 9, 2022
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Jun 9, 2022

🤖 New build scheduled with the buildbot fleet by @erlend-aasland for commit 5918fbe 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots label Jun 9, 2022
Copy link
Contributor

@animalize animalize left a comment

Please merge after a few days, maybe there is something not currently thought of.

Modules/_sqlite/statement.c Outdated Show resolved Hide resolved
- normalise switch cases
- improve NEWS entry accuracy
}

const char *p = lstrip_sql(sql_cstr);
if (p != NULL) {
is_dml = (PyOS_strnicmp(p, "insert", 6) == 0)
|| (PyOS_strnicmp(p, "update", 6) == 0)
|| (PyOS_strnicmp(p, "delete", 6) == 0)
|| (PyOS_strnicmp(p, "replace", 7) == 0);
Copy link
Contributor Author

@erlend-aasland erlend-aasland Jun 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this remark in the sqlite3_changes docs:

auxiliary changes caused by triggers, foreign key actions or REPLACE constraint resolution are not counted.

This is a separate issue, out of scope for this PR.

@erlend-aasland
Copy link
Contributor Author

@erlend-aasland erlend-aasland commented Jun 9, 2022

Thanks for reviewing, Ma Lin. Highly appreciated 🙏🏻

The buildbot run for 5918fbe completed without failures.

I'll let this PR sit around for some days to give Serhiy a chance to review. I'll merge sometime next week.

@animalize
Copy link
Contributor

@animalize animalize commented Jun 9, 2022

As I said before, I'm not a deep user of SQL. So when in very complex situations, there may be things that I can't think of.
But I will try my best to learn and understand.


parse_remaining_sql_state state = NORMAL;

for (;;) {
switch (*pos) {
case 0:
return 0;
return NULL;
Copy link
Contributor Author

@erlend-aasland erlend-aasland Jun 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: this case makes the return NULL at the end of the function unreachable. This can be easily fixed with a tiny refactor, where case 0 is removed:

lstrip_sql(const char *sql)
{
    parse_remaining_sql_state state = NORMAL;

    for (const char *pos = sql; *pos; pos++) {
        ...
    }

    return NULL;

I'll add that in a separate PR.

@animalize
Copy link
Contributor

@animalize animalize commented Jun 12, 2022

In current code, this code can't be processed correctly.

-
- INSERT INTO test(income) VALUES(?)

Maybe there are some corner cases that cannot be handled correctly as well.

IMHO, SQLite's code is simple and robust:
https://github.com/sqlite/sqlite/blob/37d4ec86bfa78c31732132b7729b8ce0e47da891/src/complete.c#L170-L178
https://github.com/sqlite/sqlite/blob/4baf43ff647ab8c6752e7fdd12efd1a979afd476/ext/rbu/sqlite3rbu.c#L2287-L2289

It uses a loop to skip comment, we can use this method instead of the state machine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants