Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tools for generating mappings_XX.h #84508

Closed
corona10 opened this issue Apr 19, 2020 · 6 comments · Fixed by #93272
Closed

Add tools for generating mappings_XX.h #84508

corona10 opened this issue Apr 19, 2020 · 6 comments · Fixed by #93272
Assignees
@corona10
Labels
3.12

Comments

@corona10
Copy link
Member

@corona10 corona10 commented Apr 19, 2020

BPO 40328
Nosy @hyeshik, @vstinner, @methane, @serhiy-storchaka, @corona10
PRs
  • #19602
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-04-19.13:13:59.400>
labels = ['3.9']
title = 'Add tools for generating mappings_XX.h'
updated_at = <Date 2020-05-25.04:14:49.537>
user = 'https://github.com/corona10'

    bugs.python.org fields:

    activity = <Date 2020-05-25.04:14:49.537>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Demos and Tools']
creation = <Date 2020-04-19.13:13:59.400>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 40328
keywords = ['patch']
message_count = 4.0
messages = ['366777', '367682', '369844', '369845']
nosy_count = 5.0
nosy_names = ['hyeshik.chang', 'vstinner', 'methane', 'serhiy.storchaka', 'corona10']
pr_nums = ['19602']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40328'
versions = ['Python 3.9']
    @corona10
    Copy link
    Member Author

    @corona10 corona10 commented Apr 19, 2020

    AFAIK, there are no tools for generating mappings_XX.h under Modules/cjkcodecs.

    It would cause a maintainable issue,
    fortunately, I got the old source from https://github.com/BackupTheBerlios/cjkpython.
    And I success to port into Python3 version and successfully reproduce to generating header file except for mappings_hk.h mappings_tw.h.

    The patch will include a generator and Unicode mapping file.
    I am not the codec expert but hopefully it will help us.

    @corona10
    Copy link
    Member Author

    @corona10 corona10 commented Apr 29, 2020

    New changeset 113feb3 by Dong-hee Na in branch 'master':
    bpo-40328: Add tool for generating cjk mapping headers (GH-19602)
    113feb3

    @methane
    Copy link
    Member

    @methane methane commented May 25, 2020

    Would you close this issue?

    @corona10
    Copy link
    Member Author

    @corona10 corona10 commented May 25, 2020

    Would you close this issue?

    mappings_hk.h mappings_tw.h are still left :)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @sorcio sorcio mentioned this issue May 3, 2022
    Open
    @sorcio
    Copy link
    Contributor

    @sorcio sorcio commented May 4, 2022

    Hey @corona10, I happened upon this for other reasons. I found that Python's big5hkscs codec does not support the same character set that modern web browsers support.

    There has been at least one version of HKSCS (2008) that updated the Big5-encoded character set, and one more recent version (2016) which only added characters to Unicode (with no Big5 mapping), but might adjust some of the existing mappings. I still need to understand if this might break compatibility.

    Is it okay if I look into this and try to make a PR later? I've already done something for mappings_hk but need to look into the compatibility, so it might take a second.

    @corona10
    Copy link
    Member Author

    @corona10 corona10 commented May 11, 2022

    @sorcio Oh thanks, PR will be welcomed :)

    @corona10 corona10 self-assigned this May 11, 2022
    sorcio added a commit to sorcio/cpython that referenced this issue May 26, 2022
    @sorcio
    pythongh-84508: tool to generate cjk traditional chinese mappings
    1fb94f7
    @sorcio sorcio mentioned this issue May 26, 2022
    Open
    @sorcio sorcio mentioned this issue May 26, 2022
    Merged
    @corona10 corona10 mentioned this issue May 28, 2022
    Merged
    corona10 added a commit that referenced this issue May 28, 2022
    @corona10
    gh-84508: Add mapping files for Korean and Japanese. (gh-93309)
    dec1e93
    corona10 added a commit that referenced this issue May 28, 2022
    @corona10
    Revert "gh-84508: Add mapping files for Korean and Japanese. (gh-93309)"
    b81a299 
    This reverts commit dec1e93.
    corona10 added a commit that referenced this issue May 29, 2022
    @corona10
    Revert "gh-84508: Add mapping files for Korean and Japanese. (gh-93309)…
    749dc4b 
    …" (#93320)

This reverts commit dec1e93.
    @corona10 corona10 added 3.12 and removed 3.9 labels May 29, 2022
    eendebakpt pushed a commit to eendebakpt/cpython that referenced this issue Jun 7, 2022
    @corona10 @eendebakpt
    Revert "pythongh-84508: Add mapping files for Korean and Japanese. (p…
    4917f8e 
    …ythongh-93309)" (python#93320)

This reverts commit dec1e93.
    corona10 pushed a commit that referenced this issue Jun 11, 2022
    @sorcio
    gh-84508: tool to generate cjk traditional chinese mappings (gh-93272)
    733e15f
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @corona10 corona10

    Labels
    3.12
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging a pull request may close this issue.

    gh-84508: tool to generate cjk traditional chinese mappings sorcio/cpython
    4 participants
    @methane @sorcio @serhiy-storchaka @corona10