Skip to content

Hi TN: percent (measure class) and time bug fixes #483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shrpawar-alt wants to merge 4 commits into
base: staging/hi_tn_v4
Choose a base branch
from
Open

Hi TN: percent (measure class) and time bug fixes #483

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
66ee78e
percent and time bug fixes
shrpawar-alt Sep 2, 2026
4e60c63
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 2, 2026
5f4b14e
merge conflict jenkinsfile resolved
shrpawar-alt Sep 8, 2026
bc55e56
Merge branch 'staging/hi_tn_v4' into hi-tn-bugfixtimeandpercent
shrpawar-alt Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ pipeline {
MR_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/03-12-24-1'
JA_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/10-17-24-1'
KO_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/06-04-25-6'
HI_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/09-04-26-0'
HI_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/09-08-26-0'
DEFAULT_TN_CACHE='/home/jenkins/TestData/text_norm/ci/grammars/06-08-23-0'
}
stages {
Expand Down
3 changes: 2 additions & 1 deletion nemo_text_processing/text_normalization/hi/data/measure/unit.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,4 +142,5 @@ mi/hr मील प्रति घंटा
mi/min मील प्रति मिनट
₹/ac रुपए प्रति एकड़
x बाई
* बाई
* बाई
% प्रतिशत
41 changes: 38 additions & 3 deletions nemo_text_processing/text_normalization/hi/taggers/measure.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,9 +228,21 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst, ordinal: GraphFst, ser
year_informal = pynini.string_map([("yr", "साल")])
year_formal = pynini.string_file(get_abs_path("data/measure/unit_year_formal.tsv"))

# All units EXCEPT year
unit_inputs_except_yr = pynini.difference(pynini.project(unit_graph, "input"), pynini.accep("yr"))
unit_graph_no_year = pynini.compose(unit_inputs_except_yr, unit_graph)
# All units EXCEPT year and percent
unit_inputs_regular = pynini.difference(
pynini.project(unit_graph, "input"), pynini.union(pynini.accep("yr"), pynini.accep("%"))

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make this a tsv file just in case we want to add in the future

)
unit_graph_no_year = pynini.compose(unit_inputs_regular, unit_graph)

percent_graph = pynini.compose(pynini.accep("%"), unit_graph)

percent_unit = (
pynutil.insert(NEMO_SPACE)
+ pynutil.insert("units: \"")
+ percent_graph
+ pynutil.insert("\"")
+ pynutil.insert(NEMO_SPACE)
)

# Load quarterly units from separate files: map (FST) and list (FSA)
quarterly_units_map = pynini.string_file(get_abs_path("data/measure/quarterly_units_map.tsv"))
Expand Down Expand Up @@ -454,14 +466,37 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst, ordinal: GraphFst, ser
+ pynutil.insert("\"")
)

graph_cardinal_percent = (
pynutil.insert("cardinal { ")
+ optional_graph_negative
+ pynutil.insert("integer: \"")
+ cardinal_graph
+ pynutil.insert("\"")
+ pynutil.insert(NEMO_SPACE)
+ pynutil.insert("}")
+ pynini.closure(delete_space, 0, 1)

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete_space is already a closure, so we might not need this, right?

+ percent_unit
)

graph_decimal_percent = (
pynutil.insert("decimal { ")
+ optional_graph_negative
+ decimal_graph
+ pynutil.insert(" }")
+ pynini.closure(delete_space, 0, 1)

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

+ percent_unit
)

address_graph = self.get_address_graph(cardinal, ordinal, serial, input_case)
structured_address_graph = self.get_structured_address_graph(cardinal, ordinal, input_case)

graph = (
pynutil.add_weight(graph_decimal, 0.1)
| pynutil.add_weight(graph_decimal_year_formal, 0.1)
| pynutil.add_weight(graph_decimal_percent, 0.1)
| pynutil.add_weight(graph_cardinal, 0.1)
| pynutil.add_weight(graph_cardinal_year_formal, 0.1)
| pynutil.add_weight(graph_cardinal_percent, 0.1)
| pynutil.add_weight(graph_cardinal_year_informal, -0.1) # Higher priority for small numbers
| pynutil.add_weight(graph_exceptions, 0.1)
| pynutil.add_weight(graph_dedh_dhai, -0.2)
Expand Down
9 changes: 5 additions & 4 deletions nemo_text_processing/text_normalization/hi/taggers/time.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,10 @@ def __init__(self, cardinal: GraphFst):
super().__init__(name="time", kind="classify")

delete_colon = pynutil.delete(":")
cardinal_graph = cardinal.digit | cardinal.teens_and_ties
delete_leading_zero = pynini.closure(pynutil.delete("0") | pynutil.delete("०"), 0, 1)
cardinal_graph = delete_leading_zero + (cardinal.digit | cardinal.teens_and_ties)

self.hours = pynutil.insert("hours: \"") + hours_graph + pynutil.insert("\" ")
self.hours = pynutil.insert("hours: \"") + delete_leading_zero + hours_graph + pynutil.insert("\" ")
self.minutes = pynutil.insert("minutes: \"") + minutes_graph + pynutil.insert("\" ")
self.seconds = pynutil.insert("seconds: \"") + seconds_graph + pynutil.insert("\" ")

Expand All @@ -73,7 +74,7 @@ def __init__(self, cardinal: GraphFst):
graph_h = self.hours + delete_colon + pynutil.delete(HI_DOUBLE_ZERO)

# Support all combinations of Devanagari and Arabic digits for dedh/dhai patterns
dedh_dhai_graph = pynini.string_map(
dedh_dhai_graph = delete_leading_zero + pynini.string_map(
[
("१:३०", HI_DEDH),
("१:30", HI_DEDH),
Expand All @@ -92,7 +93,7 @@ def __init__(self, cardinal: GraphFst):
sadhe_numbers = cardinal_graph + pynini.cross(HI_TIME_THIRTY, "")
sadhe_graph = pynutil.insert(HI_SADHE) + pynutil.insert(NEMO_SPACE) + sadhe_numbers

paune = pynini.string_file(get_abs_path("data/whitelist/paune_mappings.tsv"))
paune = delete_leading_zero + pynini.string_file(get_abs_path("data/whitelist/paune_mappings.tsv"))
paune_numbers = paune + pynini.cross(HI_TIME_FORTYFIVE, "")
paune_graph = pynutil.insert(HI_PAUNE) + pynutil.insert(NEMO_SPACE) + paune_numbers

Expand Down
4 changes: 3 additions & 1 deletion tests/nemo_text_processing/hi/data_text_normalization/test_cases_measure.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,6 @@
५ yr~पाँच साल
1.5 yr~डेढ़ साल
२.५ yr~ढाई साल
3.5 yr~साढ़े तीन साल
3.5 yr~साढ़े तीन साल

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add

50%~पचास प्रतिशत
५०%~पचास प्रतिशत
50 %~पचास प्रतिशत
५० %~पचास प्रतिशत
0%~शून्य प्रतिशत
०%~शून्य प्रतिशत
5.5%~पाँच दशमलव पाँच प्रतिशत
५.५%~पाँच दशमलव पाँच प्रतिशत
12.75%~बारह दशमलव सात पाँच प्रतिशत
१२.७५%~बारह दशमलव सात पाँच प्रतिशत
999%~नौ सौ निन्यानबे प्रतिशत
5% से 10% तक~पाँच प्रतिशत से दस प्रतिशत तक
GDP ७.५% बढ़ी~GDP सात दशमलव पाँच प्रतिशत बढ़ी
मुद्रास्फीति 5% रही।~मुद्रास्फीति पाँच प्रतिशत रही ।

100%~एक सौ प्रतिशत
५०.५० %~पचास दशमलव पाँच शून्य प्रतिशत
4 changes: 3 additions & 1 deletion tests/nemo_text_processing/hi/data_text_normalization/test_cases_time.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@
दोपहर के 3:00~दोपहर के तीन बजे
रात के १०:४८:५०~रात के दस बजकर अड़तालीस मिनट पचास सेकंड
रात के 11:50~रात के ग्यारह बजकर पचास मिनट
रात के ८:००~रात के आठ बजे
रात के ८:००~रात के आठ बजे

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add test cases to make sure we restrict mixed scripts and malformed times

0९:१५~शून्य नौ : पंद्रह
०9:15~शून्य नौ : पंद्रह
0८:४५~शून्य आठ : पैंतालीस
०8:00~शून्य आठ : शून्य शून्य
0१:३०~शून्य एक : तीस
०1:30~शून्य एक : तीस
012:30~शून्य एक दो : तीस
०१२:३०~शून्य एक दो : तीस
010:30~शून्य एक शून्य : तीस
०१०:३०~शून्य एक शून्य : तीस
023:45~शून्य दो तीन : पैंतालीस
०२३:४५~शून्य दो तीन : पैंतालीस

आज सुबह ०९:१५ बजे~आज सुबह सवा नौ बजे

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add

8:45~पौने नौ
08:45~पौने नौ
5:45~पौने छह
05:45~पौने छह

08:00~आठ बजे

@mgrafu mgrafu Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add

08:30~साढ़े आठ
०८:३०~साढ़े आठ
०८:४५~पौने नौ
01:30~डेढ़
०१:३०~डेढ़
02:30~ढाई
०२:३०~ढाई
07:15~सवा सात
०७:१५~सवा सात
09:20~नौ बजकर बीस मिनट
०९:२०~नौ बजकर बीस मिनट
00:00~शून्य बजे
००:००~शून्य बजे