are Signalcommands the only commands where package name != command name? #60

Activity

1. larsvilhuber commented on May 23, 2025

   

   larsvilhuber

   on May 23, 2025

   Member

   More actions

   Background on the Stata scanner:

   • the most current version is at https://github.com/labordynamicsinstitute/Statapackagesearch . The version in the template repo is a bit older, and not fully dynamic (on my todo list, forgot to add it to the issues, now done: Update Stata scanner #62 )
   • The better way to parse the keywords is embedded in that version. It pulls keywords from https://github.com/labordynamicsinstitute/Statapackagesearch/tree/main/auxiliary but that process, too, is not (yet) dynamic.
   • The current set of keyword-to-package mapping comes from Sergio Correia, who in turn transformed the output from the SSC mirror at https://github.com/labordynamicsinstitute/ssc-mirror/
   • I have injected some commands into the SSC mirror scripts that actually generate the raw data for this, but have not finished parsing it see https://github.com/labordynamicsinstitute/ssc-mirror/blob/main/.github/workflows/mirror.yml#L55 An example is https://github.com/labordynamicsinstitute/ssc-mirror/actions/runs/15204148070/artifacts/3183115554. This needs to be used, within a GHA workflow, to generate the right mapping file.
   • This in turn needs to be used by https://github.com/labordynamicsinstitute/stata-keyword-crosswalk/

   So the right place to fix this is in https://github.com/labordynamicsinstitute/ssc-mirror/, to there generate a list of keyword-package crosswalks as an output, and store it in the releases branch.

   Some of this has been addressed (as a dead-end branch) in the stats branch of that repo https://github.com/labordynamicsinstitute/ssc-mirror/tree/stats
2. larsvilhuber commented on May 23, 2025

   

   larsvilhuber

   on May 23, 2025

   Member

   More actions

   A further note on the Stata scanner:

   • the reason the Stata scanner is programmed in Stata was so that it would be easy to use for the target audience (classic Stata users)
   • It absolutely makes sense to use Python to do the same kind of parsing
   • I would use a "bag of words" approach, which are pre-cooked tools to parse all the words by their frequency, and possibly context.
   • You should be able to implement the logic behind the Stata Package search into Python easily.
3. floswald commented on May 23, 2025

   

   floswald

   on May 23, 2025

   Author

   More actions

   ok i see - that's what I thought/hoped. I'm looking into this right now. Not sure I understand why one needs the bag of word approach here, can you explain? Is it because some package/command names are plain english words which you would also find in a sentence? My idea was that I only look at the code files, so I don't have to worry about english prose and can forget about the context of the word. So then, if I find binscatter I can be sure that this a stata command and not an english word (well let's assume that word existed).
   I'm implemeting this in julia, there's plenty of tools around as well.
4. larsvilhuber commented on May 23, 2025

   

   larsvilhuber

   on May 23, 2025

   Member

   More actions

   There are signal words that are plain english word (table, etc.) because people have chosen that for ado files.

   IMHO, Julia is not the right tool for something that is pure text processing, but knock yourself out.
5. larsvilhuber commented on Jan 19, 2026

   

   larsvilhuber

   on Jan 19, 2026

   Member

   More actions

   @floswald Did you do this in a way that I could pull into this workflow?
6. floswald commented on Jan 19, 2026

   

   floswald

   on Jan 19, 2026

   Author

   More actions

   nope sorry...