Using ForStatementNode for AST analysis of for each statements

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Using ForStatementNode for AST analysis of for each statements

Lance Johnson
Greetings.

This email will probably contain some dumb questions buried in a jumble
of context, but I'm potentially looking at patching Flex to fix this
problem and would like to exhaust all /other/ possibilities before I do
so.  The project I am working on has already snowballed into an
avalanche, so additional complexity is something I would like to avoid.

I'm involved in a rather complicated modding project, where, due to
licensing restrictions, we are patching commercially-licensed but
undistributable code[1] by using the usual diff method, and then using
Flex's generated AST to find particular sequences of statements in
parsed code. We then patch them appropriately, and then emit the patched
ActionScript using a custom Evaluator. The evaluator in question is
based on PrettyPrinter, but is ending up being quite more exhaustive in
order to present compilable code. We use this tool to collect
information and patch repetitive statements that can be parsed with
tools like regular expressions, but need thousands of lines of
supporting code to properly keep track of context and syntax.

The problem I'm running into is that I'm having to run the Parser with
`Context.scriptAssistParsing` turned on, as I need the AST with as few
mutations as possible, and without loading includes or imports.
Unfortunately, this has resulted into running into a problem with
ForStatementNodes in this mode.

It appears that, while `for each` statements are parsed, there is no
indication applied in NodeFactory to the ForStatementNode to indicate
that the Node is a `for each` loop /in scriptAssistParsing mode/. In
addition, if I were to turn scriptAssistParsing off, there would be
significant mutations applied.  In NodeFactory.java, it appears like
there were plans made to introduce a boolean (is_each?) to mark this,
but it never materialized.

Is there another way to determine this information, or will I need to
patch Flex itself to expose it?  If this is something that should be
added to the upstream project, what would be the most agreeable means to
implement it?  Right now, I simply plan on adding a public boolean
is_each to ForStatementNode, and updating NodeFactory accordingly.

Thanks in advance,

Lance Johnson

1: The owners of the code in question have released it on GitHub, but
the license terms forbid distribution of modified code or binaries, so
we distribute our mod as patches, instead.  There's also talk of more
permissive licensing in the future. Maybe.
Reply | Threaded
Open this post in threaded view
|

Re: Using ForStatementNode for AST analysis of for each statements

Alex Harui-2
Hi,

I do not know the Flex MXMLC compiler very well at all, and I'm not sure anyone else does either at this point.

The next generation compiler (was known as Falcon) might be worth a try.  A few more of us do know this compiler better, and you might find that its AST has what you need.   The Apache Royale project has the latest version of the Falcon compiler so inquire on [hidden email] if you interested.

HTH,
-Alex

On 5/8/19, 10:15 PM, "Lance Johnson" <[hidden email]> wrote:

    Greetings.
   
    This email will probably contain some dumb questions buried in a jumble
    of context, but I'm potentially looking at patching Flex to fix this
    problem and would like to exhaust all /other/ possibilities before I do
    so.  The project I am working on has already snowballed into an
    avalanche, so additional complexity is something I would like to avoid.
   
    I'm involved in a rather complicated modding project, where, due to
    licensing restrictions, we are patching commercially-licensed but
    undistributable code[1] by using the usual diff method, and then using
    Flex's generated AST to find particular sequences of statements in
    parsed code. We then patch them appropriately, and then emit the patched
    ActionScript using a custom Evaluator. The evaluator in question is
    based on PrettyPrinter, but is ending up being quite more exhaustive in
    order to present compilable code. We use this tool to collect
    information and patch repetitive statements that can be parsed with
    tools like regular expressions, but need thousands of lines of
    supporting code to properly keep track of context and syntax.
   
    The problem I'm running into is that I'm having to run the Parser with
    `Context.scriptAssistParsing` turned on, as I need the AST with as few
    mutations as possible, and without loading includes or imports.
    Unfortunately, this has resulted into running into a problem with
    ForStatementNodes in this mode.
   
    It appears that, while `for each` statements are parsed, there is no
    indication applied in NodeFactory to the ForStatementNode to indicate
    that the Node is a `for each` loop /in scriptAssistParsing mode/. In
    addition, if I were to turn scriptAssistParsing off, there would be
    significant mutations applied.  In NodeFactory.java, it appears like
    there were plans made to introduce a boolean (is_each?) to mark this,
    but it never materialized.
   
    Is there another way to determine this information, or will I need to
    patch Flex itself to expose it?  If this is something that should be
    added to the upstream project, what would be the most agreeable means to
    implement it?  Right now, I simply plan on adding a public boolean
    is_each to ForStatementNode, and updating NodeFactory accordingly.
   
    Thanks in advance,
   
    Lance Johnson
   
    1: The owners of the code in question have released it on GitHub, but
    the license terms forbid distribution of modified code or binaries, so
    we distribute our mod as patches, instead.  There's also talk of more
    permissive licensing in the future. Maybe.
   

Reply | Threaded
Open this post in threaded view
|

Re: Using ForStatementNode for AST analysis of for each statements

Lance Johnson
I may take a look later on.  I already have quite a bit of code invested
in Flex, and the code I'm parsing is compiled with Flex, so I'm stuck
with it, for now.

I went ahead and made the necessary changes, ended up being fairly
simple.  Unfortunately, my editor "fixed" a bunch of trailing whitespace
abnormalities, so it may be a bit before I can produce a "clean" diff.
Changes are at
<https://gitlab.com/VentureGuy/flex-sdk/commit/10dff5661cef6eaeda2f68e51f821aaa8daf874b>,
in case someone else is in the same boat and comes by this thread via
Google.

Thanks for your time.

On 5/9/2019 9:03 PM, Alex Harui wrote:

> Hi,
>
> I do not know the Flex MXMLC compiler very well at all, and I'm not sure anyone else does either at this point.
>
> The next generation compiler (was known as Falcon) might be worth a try.  A few more of us do know this compiler better, and you might find that its AST has what you need.   The Apache Royale project has the latest version of the Falcon compiler so inquire on [hidden email] if you interested.
>
> HTH,
> -Alex
>
> On 5/8/19, 10:15 PM, "Lance Johnson" <[hidden email]> wrote:
>
>      Greetings.
>      
>      This email will probably contain some dumb questions buried in a jumble
>      of context, but I'm potentially looking at patching Flex to fix this
>      problem and would like to exhaust all /other/ possibilities before I do
>      so.  The project I am working on has already snowballed into an
>      avalanche, so additional complexity is something I would like to avoid.
>      
>      I'm involved in a rather complicated modding project, where, due to
>      licensing restrictions, we are patching commercially-licensed but
>      undistributable code[1] by using the usual diff method, and then using
>      Flex's generated AST to find particular sequences of statements in
>      parsed code. We then patch them appropriately, and then emit the patched
>      ActionScript using a custom Evaluator. The evaluator in question is
>      based on PrettyPrinter, but is ending up being quite more exhaustive in
>      order to present compilable code. We use this tool to collect
>      information and patch repetitive statements that can be parsed with
>      tools like regular expressions, but need thousands of lines of
>      supporting code to properly keep track of context and syntax.
>      
>      The problem I'm running into is that I'm having to run the Parser with
>      `Context.scriptAssistParsing` turned on, as I need the AST with as few
>      mutations as possible, and without loading includes or imports.
>      Unfortunately, this has resulted into running into a problem with
>      ForStatementNodes in this mode.
>      
>      It appears that, while `for each` statements are parsed, there is no
>      indication applied in NodeFactory to the ForStatementNode to indicate
>      that the Node is a `for each` loop /in scriptAssistParsing mode/. In
>      addition, if I were to turn scriptAssistParsing off, there would be
>      significant mutations applied.  In NodeFactory.java, it appears like
>      there were plans made to introduce a boolean (is_each?) to mark this,
>      but it never materialized.
>      
>      Is there another way to determine this information, or will I need to
>      patch Flex itself to expose it?  If this is something that should be
>      added to the upstream project, what would be the most agreeable means to
>      implement it?  Right now, I simply plan on adding a public boolean
>      is_each to ForStatementNode, and updating NodeFactory accordingly.
>      
>      Thanks in advance,
>      
>      Lance Johnson
>      
>      1: The owners of the code in question have released it on GitHub, but
>      the license terms forbid distribution of modified code or binaries, so
>      we distribute our mod as patches, instead.  There's also talk of more
>      permissive licensing in the future. Maybe.
>      
>