FWD » Core Development » Conversion Tools

We have 3 modes for specifying the source files to process in the ConversionDriver.

The result causes issues:

1. In complex projects, one must often specify more files than needed.
   • In -f mode, it would be better if you could use a specification to set a default so that only a small number of exceptions could be written. Instead one must list every file in the entire project. It is a pain to create in the first place and then it is even more work to maintaining over time (as new files are added/removed in the project).
   • In -x mode, the addition of the file spec is really useful. The problem comes when one wants to include a small number of files from an otherwise fully excluded directory tree. A classic example is the case where you just want to include small number of Possenet/ADM2 files but otherwise exclude the majority of it. Look at this (from a real project):

     ./abl/possenet/src/adebuild/*
     ./abl/possenet/src/adecomm/_*
     ./abl/possenet/src/adecomm/*.def
     ./abl/possenet/src/adecomm/convcp.p
     ./abl/possenet/src/adecomm/xmlschema.p
     ./abl/possenet/src/adecomp/*
     ./abl/possenet/src/adedesk/*
     ./abl/possenet/src/adedict/*
     ./abl/possenet/src/adeedit/*
     ./abl/possenet/src/adeicon/*
     ./abl/possenet/src/aderes/*
     ./abl/possenet/src/aderesc/*
     ./abl/possenet/src/adeshar/*
     ./abl/possenet/src/adetran/*
     ./abl/possenet/src/adettdb/*
     ./abl/possenet/src/adeuib/*
     ./abl/possenet/src/adeweb/*
     ./abl/possenet/src/adexml/*
     ./abl/possenet/src/adm/*
     ./abl/possenet/src/adm2/a*.*
     ./abl/possenet/src/adm2/b*.*
     ./abl/possenet/src/adm2/ca*.*
     ./abl/possenet/src/adm2/cob*.*
     ./abl/possenet/src/adm2/com*.*
     ./abl/possenet/src/adm2/cons*.*
     ./abl/possenet/src/adm2/containr.ado
     ./abl/possenet/src/adm2/containr.cld
     ./abl/possenet/src/adm2/custom/*
     ./abl/possenet/src/adm2/da*.*
     ./abl/possenet/src/adm2/de*.*
     ./abl/possenet/src/adm2/dynb*.*
     ./abl/possenet/src/adm2/dync*.*
     ./abl/possenet/src/adm2/dyndata.ado
     ./abl/possenet/src/adm2/dynf*.*
     ./abl/possenet/src/adm2/dynl*.*
     ./abl/possenet/src/adm2/dynp*.*
     ./abl/possenet/src/adm2/dynr*.*
     ./abl/possenet/src/adm2/dyns*.*
     ./abl/possenet/src/adm2/dynt*.*
     ./abl/possenet/src/adm2/dynw*.*
     ./abl/possenet/src/adm2/e*.*
     ./abl/possenet/src/adm2/fetchc*.*
     ./abl/possenet/src/adm2/fetchdata.ado
     ./abl/possenet/src/adm2/fetchde*.*
     ./abl/possenet/src/adm2/fetchrows.ado
     ./abl/possenet/src/adm2/fi*.*
     ./abl/possenet/src/adm2/fo*.*
     ./abl/possenet/src/adm2/g*.*
     ./abl/possenet/src/adm2/i*.*
     ./abl/possenet/src/adm2/image/*
     ./abl/possenet/src/adm2/j*.*
     ./abl/possenet/src/adm2/l*.*
     ./abl/possenet/src/adm2/m*.*
     ./abl/possenet/src/adm2/p*.*
     ./abl/possenet/src/adm2/q*.*
     ./abl/possenet/src/adm2/r*.*
     ./abl/possenet/src/adm2/sa*.*
     ./abl/possenet/src/adm2/sb*.*
     ./abl/possenet/src/adm2/sc*.*
     ./abl/possenet/src/adm2/sd*.*
     ./abl/possenet/src/adm2/se*.*
     ./abl/possenet/src/adm2/smart.ado
     ./abl/possenet/src/adm2/smart.cld
     ./abl/possenet/src/adm2/smr*.*
     ./abl/possenet/src/adm2/support/_*.*
     ./abl/possenet/src/adm2/support/b*.*
     ./abl/possenet/src/adm2/support/c*.*
     ./abl/possenet/src/adm2/support/d*.*
     ./abl/possenet/src/adm2/support/f*.*
     ./abl/possenet/src/adm2/support/n*.*
     ./abl/possenet/src/adm2/support/p*.*
     ./abl/possenet/src/adm2/support/r*.*
     ./abl/possenet/src/adm2/support/s*.*
     ./abl/possenet/src/adm2/support/t*.*
     ./abl/possenet/src/adm2/support/u*.*
     ./abl/possenet/src/adm2/support/vb*.*
     ./abl/possenet/src/adm2/support/vc*.*
     ./abl/possenet/src/adm2/support/vie*.*
     ./abl/possenet/src/adm2/support/visuald.ado
     ./abl/possenet/src/adm2/template/*
     ./abl/possenet/src/adm2/t*.*
     ./abl/possenet/src/adm2/u*.*
     ./abl/possenet/src/adm2/view*.*
     ./abl/possenet/src/adm2/visp*.*
     ./abl/possenet/src/adm2/visual.ado
     ./abl/possenet/src/adm2/visual.cld
     ./abl/possenet/src/adm2/w*.*
     ./abl/possenet/src/adm2/x*.*
     ./abl/possenet/src/as4dict/*
     ./abl/possenet/src/dynamics/af/app/*
     ./abl/possenet/src/dynamics/af/bmp/*
     ./abl/possenet/src/dynamics/af/cod/*
     ./abl/possenet/src/dynamics/af/cod2/afa*.*
     ./abl/possenet/src/dynamics/af/cod2/afc*.*
     ./abl/possenet/src/dynamics/af/cod2/afg*.*
     ./abl/possenet/src/dynamics/af/cod2/afm*.*
     ./abl/possenet/src/dynamics/af/cod2/afp*.*
     ./abl/possenet/src/dynamics/af/cod2/afs*.*
     ./abl/possenet/src/dynamics/af/cod2/aftemc*.*
     ./abl/possenet/src/dynamics/af/cod2/afteml*.*
     ./abl/possenet/src/dynamics/af/cod2/aftemsuspd.ado
     ./abl/possenet/src/dynamics/af/cod2/aftemw*.*
     ./abl/possenet/src/dynamics/af/cod2/d*.*
     ./abl/possenet/src/dynamics/af/cod2/f*.*
     ./abl/possenet/src/dynamics/af/cod2/g*.*
     ./abl/possenet/src/dynamics/af/cod2/p*.*
     ./abl/possenet/src/dynamics/af/cod2/r*.*
     ./abl/possenet/src/dynamics/af/cod2/s*.*
     ./abl/possenet/src/dynamics/af/obj2/*
     ./abl/possenet/src/dynamics/af/rep/*
     ./abl/possenet/src/dynamics/af/sup/*
     ./abl/possenet/src/dynamics/af/sup2/*
     ./abl/possenet/src/dynamics/afr*.*
     ./abl/possenet/src/dynamics/as*.*
     ./abl/possenet/src/dynamics/db/*
     ./abl/possenet/src/dynamics/i*.*
     ./abl/possenet/src/dynamics/icf/*
     ./abl/possenet/src/dynamics/install/*
     ./abl/possenet/src/dynamics/p*.*
     ./abl/possenet/src/dynamics/ry/*
     ./abl/possenet/src/dynamics/scm/*
     ./abl/possenet/src/prodict/*
     ./abl/possenet/src/prohelp/*
     ./abl/possenet/src/prores/*
     ./abl/possenet/src/protools/*
     ./abl/possenet/src/template/*
     ./abl/possenet/src/web/*
     ./abl/possenet/src/web2/*
     ./abl/possenet/src/webedit/*
     ./abl/possenet/src/webspeed/*
     ./abl/possenet/src/webtools/*
     ./abl/possenet/src/webutil/*
     ./abl/possenet/src/workshop/*
     ./abl/possenet/src/wrappers/*
     

2. The scripting for projects is very complex and often broken in some build/conversion targets. The standard project template has 3 front end targets, 3 main conversion targets etc... (one for -s, one for -f and one for -x).
3. Very confusing.
   • It is all very confusing for users, especially in any scenario where either or both of the file/ignore lists exists.
   • Notice that the possenet exclude list above is very long and has none of the actual included files in it. This means that it is error prone and it is hard to reason about.

We only need a single mode By implementing a single approach all of the above problems will be gone. The idea:

1. Same 3 parameter syntax as -x:
   • top_level_src_dir
   • file_spec
   • list_filename
2. We should default the file_spec to (*.[pPwW]|*.cls). The user should be able to optionally override this with their own spec.
3. The only required parameter would be top_level_src_dir. I don't think it is reasonable to default this (yet).
4. If the optional list_filename is specified, then each line would be in the format <directive> <file_or_path_spec>.
   • directive would be I (include the spec) or X (exclude the spec), case-insensitive.
   • file_or_path_spec would be the same syntax used by -x mode today, except it would be interpreted based on the directive.
   • The is an implicit I <command_line_or_default_file_spec at the top of the file.
   • The file is processed top to bottom, with each directive potentially editing the list (adding or removing files).

The above example of Possenet files would be re-written as this condensed version:

X ./abl/possenet/*
I ./abl/possenet/src/adecomm/as-utils.w
I ./abl/possenet/src/adecomm/get-user.p
I ./abl/possenet/src/adm2/containr.p
I ./abl/possenet/src/adm2/dyndata.w
I ./abl/possenet/src/adm2/fetchdata.p
I ./abl/possenet/src/adm2/fetchrows.p
I ./abl/possenet/src/adm2/smart.p
I ./abl/possenet/src/adm2/support/visuald.w
I ./abl/possenet/src/adm2/visual.p
I ./abl/possenet/src/dynamics/af/cod2/aftemsuspd.w

Greg, to make something clear: the X and I filters will work only on the set of files determined from the D, N, F operations, right? So, if an I filter references an exact file which is not part of the D + N + F file set, then it will not be included.

The 'dry run' (-l) mode will be used to test compatibility between a -Z mode and an existing whitelist (-f) or blacklist (-x) mode, so we know the same files are converted, for existing projects.

The changes are in 3821c/13778 for both this task and #6253.