FWD » Core Development » User Interface

The following is copied from #3767.

From Marius:

A .wrx file is MS CFB file (https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-cfb/53989ce4-7b05-4f8d-829b-d08d6148375b) a general-purpose file format that provides a file-system-like structure within a file for the storage of arbitrary, application-specific streams of data. This low level format is used by other Microsoft documents like .doc (Word document), .msg (mail message), etc. For example you can try to rename a .wrx file as .doc file and you will see that Word and LibreOffice Writer are able to open the file.

The problem is that the content is in binary format specific for each application .wrx, .msg, .doc. .msg file structure is documented by MS but I'm afraid the .wrx binary format content is created by Progress Software and not documented. https://knowledgebase.progress.com/articles/Article/What-do-wrx-files-contain

Some time ago I created a Python script able to parse .msg files and extract mail properties and attachments.

From Greg:

A .wrx file is MS CFB file

This is an interesting detail. I had assumed it was something completely proprietary to the 4GL. Looking through the specification, I note there is this "OLE Property Sets" specification:

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-oleps/bf7aeae8-c47a-4939-9f45-700158dac3bc

I wonder if Progress just implemented such an approach.

All settings I did at runtime are stored perhaps inside .wrx file and are loaded at runtime during initialization!

Yes, this is something we need to resolve. I agree that parsing the .wrx file is the best approach. I do prefer to convert any such design-time features into static code references. In other words, I want to eliminate the .wrx file itself after conversion. Since one cannot edit that file easily (or at all) and it is difficult (impossible?) to read the settings later, it is a terrible way to store settings. It is much better to leave this as code that can be edited and read.

The down side is the scenario where there are more than one .wrx conditionally used in the same LoadControls. I've never seen such an approach, though it should be possible. In such a case we would also have to write the settings code in a conditional manner. For now we can ignore this problem.

Please look into the .wrx format and see if you can decode it. Perhaps there is some generic tool for viewing/parsing these files that can at least be used to understand the format. I'm going to create a separate task for this so that the findings will be public.

Any parsing we do would need to be using Java. Each of the properties set from the .wrx would be converted into OCX setter calls in the converted code. The .wrx would not be used at runtime.

Greg Shah wrote:

Any parsing we do would need to be using Java. Each of the properties set from the .wrx would be converted into OCX setter calls in the converted code. The .wrx would not be used at runtime.

I see a possible trouble here. I suspect, that the .wrx files contain setters for all the OCX properties, which were exposed by the original OCX at the moment the .wrx file was created, most of these being just default values, which need not be set.

At the moment, is is safe to not implement some of original OCX properties, if applications does not set them. Implementing the proposed .wrx file processing will broke these OCX implementations.

To solve this, the conversion program may statically check the implemented in FWD OCX interface and do not convert the missing properties, probably, just reporting them.

Moreover, I suppose most of the real .wrx files contains only the default values, so no code at all should be generated for these files.

To make it happen, we should let the conversion program to know all the default property values. These we can, for example, define in the ComProperty annotaions for property setters.

To make it happen, we should let the conversion program to know all the default property values. These we can, for example, define in the ComProperty annotaions for property setters.

Good idea.

Looking over properties data in .wrx file I didn't found any property name as a string! The problem is how we can associate a value to a property?

Take a look at the OLE Property Sets specification. COM is based on OLE. Each property is represented by a property ID and the string name is optional. I guess that OLE has some way to map these IDs to the specific properties being referenced. Each control is just a DLL after all, with various function entry points (COM methods) and exported memory locations (COM properties) which are defined as the external API of the DLL. It is common in Windows DLLs to assign ordinal values to exported entries. Perhaps the property ID is somehow related to that ordinal.

Greg Shah wrote:

Looking over properties data in .wrx file I didn't found any property name as a string! The problem is how we can associate a value to a property?

Take a look at the OLE Property Sets specification. COM is based on OLE. Each property is represented by a property ID and the string name is optional. I guess that OLE has some way to map these IDs to the specific properties being referenced. Each control is just a DLL after all, with various function entry points (COM methods) and exported memory locations (COM properties) which are defined as the external API of the DLL. It is common in Windows DLLs to assign ordinal values to exported entries. Perhaps the property ID is somehow related to that ordinal.

If I remember this correctly, the individual COM methods and properties are not exported as DLL entries. Only a few entries are required:

BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID )
STDAPI DllCanUnloadNow(void)
STDAPI DllGetClassObject(REFCLSID rclsid, REFIID riid, LPVOID* ppv)
STDAPI DllRegisterServer(void)
STDAPI DllUnregisterServer(void)

And I suppose, the DllGetClassObject returns the interface descriptions we are looking for.

Greg Shah wrote:

Looking over properties data in .wrx file I didn't found any property name as a string! The problem is how we can associate a value to a property?

Take a look at the OLE Property Sets specification. COM is based on OLE. Each property is represented by a property ID and the string name is optional. I guess that OLE has some way to map these IDs to the specific properties being referenced. Each control is just a DLL after all, with various function entry points (COM methods) and exported memory locations (COM properties) which are defined as the external API of the DLL. It is common in Windows DLLs to assign ordinal values to exported entries. Perhaps the property ID is somehow related to that ordinal.

COM methods (and properties) can be invoked dynamically through the optional IDispatch interface. If you look at its Invoke method, it declares a DISPID parameter, which is an ordinal value of the method/prop being invoked, see https://docs.microsoft.com/en-us/windows/win32/api/oaidl/nf-oaidl-idispatch-invoke. The dispid is defined in a type library AFAIK. Also there is another method IDispatch::GetIDsOfNames which can translate the ids to names, I guess it goes to the type library and reads the info there.

I wonder if this dispid is somehow stored in the wrx file.

This is very good work!

Your point about the Appbuilder made me think about the fact that the original Appbuilder code is open source written in the 4GL. My idea was to check if the binary .wrx writing code was included. I looked in the Possenet code for references to .wrx files. In src/adeuib/_recreat.p we can see this code:

        /* Save the control to the wrx file using the control-frame name */
        f_U._COM-HANDLE:SaveControls(tempBinaryFile, f_U._NAME).

I believe that SaveControls is a 4GL language feature (probably written in C like the rest of the 4GL) which saves off the binary .wrx file. It seems to be the design-time mirror image to the LoadControls method that is also on the CONTROL-FRAME.

Anyway, it seems like you already have enough information to parse the parts of the file that we care about. That is great!

In FWD we cannot depend on OCX instance nor in Type Library. IMO one possible solution is to keep OCX properties metadata in a XML structure like:

...

This file should be created once and used whenever we need to decode values for same OCX type (CLSID). The link between metadata from XML file and .wrx data is the OCX CLSID which is unique. Here we can keep among the property number and type attributes a skip attribute which specify if the property should be set or not. That is, we decode property value but we don't set the property value, we just skip over.

I like this idea. We would have to write a tool (using native code, similar to src/native/get_text_metrics.c) which would capture this based on some list of OCX controls. Then we would store the result in a repo.

I don't quite understand the skip idea. I think it is more important to know the default value for each property. If the value encoded in the wrx is different from the default, then we would write the property setter out into the Java code, otherwise we ignore it.

Greg Shah wrote:

[...]

Anyway, it seems like you already have enough information to parse the parts of the file that we care about. That is great!
[...]

I like this idea. We would have to write a tool (using native code, similar to src/native/get_text_metrics.c) which would capture this based on some list of OCX controls. Then we would store the result in a repo.
[...]

I don't quite understand the skip idea. I think it is more important to know the default value for each property. If the value encoded in the wrx is different from the default, then we would write the property setter out into the Java code, otherwise we ignore it.

1. We are close to have enough information to decode OCX property values from .wrx files. The name of the OCX control is also important even though is not an OCX property. So far we have some conclusions as a result of analyzing data for PSTimer OCX only. This is not enough that's why I'm going to analyse data for other OCX as well, for example ProgressBar.

2. Reading data from (scanning) .wrx files is very easy to implement. Using Apache POI (https://poi.apache.org/) we can do that in just a few lines of code. Apache POI is an open source project available under APACHE LICENSE, VERSION 2.0

3. Yes, we have to create a tool to extract metadata from OCX TLB and build the XML file, otherwise we should do that by hand. This tool will work only in Windows OS, we cannot do that in Linux. Nowadays almost all in Windows are .NET based. I have no idea on how we can do that in C code! Also I saw some posts which claims that using Powershell it is possible to analyze COM objects. A Powershell script would be better because is easy to maintain and is ready to run, no compile is needed. The last Windows C program I wrote is the p2j winspawn.c, 5-6 years ago!!!

4. Once we have the XML metadata, parsing data should be a trivial implementation. I proposed the skip attribute for unimplemented properties. On the other hand for unimplemented properties, conversion rules will add a noop-com annotation which in turn will generate a no operation COM chain, so we don't need a skip attribute for this purpose.

Using Apache POI (https://poi.apache.org/) we can do that in just a few lines of code. Apache POI is an open source project available under APACHE LICENSE, VERSION 2.0

This is acceptable. You will have to update the gradle dependencies to add this.

This tool will work only in Windows OS, we cannot do that in Linux. Nowadays almost all in Windows are .NET based. I have no idea on how we can do that in C code! Also I saw some posts which claims that using Powershell it is possible to analyze COM objects. A Powershell script would be better because is easy to maintain and is ready to run, no compile is needed. The last Windows C program I wrote is the p2j winspawn.c, 5-6 years ago!!!

We already have some native code that does this. For example, you can see in src/native/comauto_win.c, there are C functions like getComName(int id), getComProperty(int id, String name), getMethodType(int id). I'm hoping that with our existing code with uses the IDispatch API, that you can see how GetTypeInfo() and other such helpers work.

I prefer to extend this code with any needed inspection functions and then write Java code to do the rest.

Why Value property is not available for change at design time? What is criteria used by Application Builder to decide not to expose this property at design time?

I think Value is treated as a special property. From IDispatch::GetIDsOfNames():

An IDispatch implementation can associate any positive integer ID value with a given name. Zero is reserved for the default, or Value property; –1 is reserved to indicate an unknown name; and other negative values are defined for other purposes.

The 4GL may simply only deal with non-zero property ids. On the other hand, the type library tool reports Value as property id 5 so I don't know. Perhaps the properties with the name "Value" are also excluded as special (not to be set at design time).

It's easy to traverse and extract data for all these structures except for OCX:DATA which is the most complex structure and looks different for each OCX.
Unfortunately it's impossible to find a logic behind the structure of OCX:DATA block! There are a lot of unknown data in this block which make impossible to "guess" the real structure.

I wonder if the 1C 00 00 00 at AB is the length of the property data (from AF through CA), just before the FFFFFFFFFFFFFFFF at CB. Then again, this doesn't appear in the PSTimer data so it is probably just me imagining patterns where none exist.

I guess one of the following is true:

1. The structure of the OCX:DATA section can be deduced from the property type info that can be inspected via the COM APIs; OR
2. The structure is encoded inside the .wrx itself. The only obvious possibilities of this is the data from 93 through AA and/or BB through C6 and/or the data from D3 through E6. Everything else seems to be accounted for.

Considering that the property ids are not encoded in the data and both the size and order of the OCX:DATA block is fixed, I suspect it can be deduced from the exposed COM API (e.g. inspecting the type library).

Why Application Builder combine those values? Which criteria is used to decide how to combine values and for which properties?

Are you sure that the Enabled property is stored in that same bit field? All the other values stored there are part of the same set of MS common control constants. In other words, the type library for the common controls probably exports this as a single bit field that has multiple enums defined as "sections". I suspect that the masks and encoding map directly to how the type library reports the properties structure.

As a next step, please dig in to the type library APIs (the ITypeLib and ITypeInfo seem most useful). Writing some code to use these APIs may explain some things that are otherwise quite strange right now.

Greg Shah wrote:

Why Value property is not available for change at design time? What is criteria used by Application Builder to decide not to expose this property at design time?

The following is an excerpt from here: https://documentation.progress.com/output/ua/OpenEdge_latest/dvpin/understanding-design-time-and-runtime-properties.html

Understanding design time and runtime properties

To customize the definition for an ActiveX control, you must change the values of control properties in the AppBuilder at design time. Each ActiveX control supports a specific list of properties that you can set at design time (design time properties) and another list of properties that you can read or write at runtime (run-time properties). (You can set some properties at both design time and runtime.)

To modify design time properties, you generally require a license from the control vendor. The vendor typically provides this license to application developers (as opposed to application end-users) for installation with the control.

The AppBuilder provides access to all available design time properties using the OCX Property Editor window. This window contains all design time properties including the extended properties that ABL adds. This is the only way to set values for properties writable only at design time.

To modify run-time properties, you write ABL code that sets the property value using the component handle to the control.

The Value property of ProgressBar is a runtime property, you cannot set it neither in AppBuilder not it the "native" property editor in VisualStudio.

The appbuilder code (see possenet in the src/adeuib/ directory) itself uses an OCX control for the "Property Editor" (and probably for some other parts too). I wonder if this OCX is from Microsoft. Anyway, the reliance upon external native code may be one reason for the appbuilder to be quite unstable.

Greg Shah wrote:

The appbuilder code (see possenet in the src/adeuib/ directory) itself uses an OCX control for the "Property Editor" (and probably for some other parts too).

Can't we use the OCX to parse WRX files?

Hynek Cihlar wrote:

Greg Shah wrote:

The appbuilder code (see possenet in the src/adeuib/ directory) itself uses an OCX control for the "Property Editor" (and probably for some other parts too).

Can't we use the OCX to parse the WRX files?

WRX files are from the OpenEdge domain, OCX controls know nothing about the WRX.

We can probably extract the data from a WRX file using the following steps:

1. In AppBuilder, create a control for the same type of OCX, save the control along with a WRX file.
2. Replace the WRX file by one we want to extract data from
3. Re-open the control with AppBuilder, see the property values.

Vladimir Tsichevski wrote:

Hynek Cihlar wrote:

Greg Shah wrote:

The appbuilder code (see possenet in the src/adeuib/ directory) itself uses an OCX control for the "Property Editor" (and probably for some other parts too).

Can't we use the OCX to parse the WRX files?

WRX files are from the OpenEdge domain, OCX controls know nothing about the WRX.

I meant the specific OCX control Greg mentions above. If it is used for showing/editing OCX properties, we could use it, too.

Hynek Cihlar wrote:

Vladimir Tsichevski wrote:

Hynek Cihlar wrote:

Greg Shah wrote:

The appbuilder code (see possenet in the src/adeuib/ directory) itself uses an OCX control for the "Property Editor" (and probably for some other parts too).

Can't we use the OCX to parse the WRX files?

WRX files are from the OpenEdge domain, OCX controls know nothing about the WRX.

I meant the specific OCX control Greg mentions above. If it is used for showing/editing OCX properties, we could use it, too.

Which exactly OCX control is it?

Vladimir Tsichevski wrote:

Which exactly OCX control is it?

See Greg's note #4438-17. I just had a quick look in the possenet sources so I can't give you good answer ATM.

Hynek Cihlar wrote:

Vladimir Tsichevski wrote:

Which exactly OCX control is it?

See Greg's note #4438-17. I just had a quick look in the possenet sources so I can't give you good answer ATM.

If this OCX is from Microsoft, when it cannot be used to read WRX files.

The OCX properties are indeed handled by a COM component, but this comes with OpenEdge. It resides in prox.dll and the following is its type library.

prox.dll type libraryprox.dll type library

// Generated .IDL file (by the OLE/COM Object Viewer)
// 
// typelib filename: prox.dll

[
  uuid(85E391ED-11A2-46B1-94AD-D4FD17266CB3),
  version(11.0),
  helpstring("ProX Library"),
  custom(DE77BA64-517C-11D1-A2DA-0000F8773CE9, 117441067),
  custom(DE77BA63-517C-11D1-A2DA-0000F8773CE9, 1445049543),
  custom(DE77BA65-517C-11D1-A2DA-0000F8773CE9, "Created by MIDL version 7.00.0555 at Fri Oct 16 22:39:02 2015
")

]
library ProX
{
    // TLib : OLE Automation : {00020430-0000-0000-C000-000000000046}
    importlib("stdole2.tlb");

    // Forward declare all types defined in this typelib
    dispinterface IProIDE;
    dispinterface IControlDef;
    dispinterface IControls;
    dispinterface IProControlFrame;
    dispinterface IProControlFrameEvents;
    dispinterface IEvents;
    dispinterface IProEvent;
    dispinterface IParams;
    dispinterface IParam;
    dispinterface IExtendedObject;
    dispinterface IExtendedEvents;
    dispinterface IProgressRuntime;

    [
      uuid(CB1A21E1-F3A8-4D92-9FE9-1E870147FEFD)
    ]
    dispinterface IProIDE {
        properties:
            [id(0x00000001)            
]
            VARIANT_BOOL PropertyEditorVisible;
            [id(0x00000002)            
]
            BSTR HelpFile;
        methods:
            [id(0x00000003)]
            void EditProperties(IDispatch* pControl);
            [id(0x00000004)]
            void AboutBox();
            [id(0x00000005)]
            IDispatch* GetControl(long hParentWnd);
            [id(0x00000006)]
            IDispatch* GetControlEvents(IDispatch* pControl);
            [id(0x00000007)]
            BSTR VBXToOCXClassID(
                            BSTR bstrIniString, 
                            BSTR bstrControlName);
            [id(0x00000008)]
            void ConvertWBXToWRX(
                            BSTR WBXFileName, 
                            BSTR WRXFileName);
            [id(0x00000009)]
            void ClearProperties();
            [id(0x0000000a)]
            void GetPropertyEditorPosition(
                            long* left, 
                            long* top, 
                            long* width, 
                            long* height);
            [id(0x0000000b)]
            void SetPropertyEditorPosition(
                            long left, 
                            long top, 
                            long width, 
                            long height);
    };

    [
      uuid(FAB261B7-C9D9-4FAA-8FB4-35EE3E88D35C)
    ]
    coclass PROIDE {
        [default] dispinterface IProIDE;
    };

    [
      uuid(D83BB36D-690D-4BC0-B4E4-9E87DCA78995)
    ]
    dispinterface IControlDef {
        properties:
            [id(0x00000001)            
]
            BSTR ClassID;
            [id(0x00000002), readonly            
]
            long Bitmap;
            [id(0x00000003)            
]
            BSTR Name;
            [id(0x00000004)            
]
            BSTR ShortName;
            [id(0x00000005)            
]
            BSTR Path;
            [id(0x00000006)            
]
            BSTR HelpFile;
            [id(0x00000007)            
]
            long HelpContext;
            [id(0x00000008)            
]
            BSTR Version;
        methods:
            [id(0x00000009)]
            void SaveDownBitmap(BSTR szFileName);
            [id(0x0000000a)]
            void SaveUpBitmap(BSTR szFileName);
    };

    [
      uuid(B740A502-01B8-448B-9026-CE193FE3C82E)
    ]
    coclass CONTROLDEF {
        [default] dispinterface IControlDef;
    };

    [
      uuid(FFF7A3FA-17A6-4709-98D0-A01A6E6F7F36)
    ]
    dispinterface IControls {
        properties:
            [id(0x00000001), readonly, hidden            
]
            IUnknown* NewEnum;
            [id(0x00000002), readonly            
]
            long Count;
        methods:
            [id(0x00000003)]
            IDispatch* Item(VARIANT Item);
            [id(0x00000004), hidden]
            void SetFrame(long pFrame);
    };

    [
      uuid(6FDB79D2-E2DB-4A6A-AEE6-70C27161A678)
    ]
    coclass CONTROLS {
        [default] dispinterface IControls;
    };

    [
      uuid(FF5BF0E9-B9BD-4C46-B988-F70462DF18C3)
    ]
    dispinterface IProControlFrame {
        properties:
            [id(00000000)            
]
            BSTR _Name;
            [id(0x00000001)            
]
            BSTR Name;
            [id(0x00000002)            
]
            VARIANT_BOOL DesignMode;
            [id(0x00000003)            
]
            VARIANT_BOOL Dirty;
            [id(0x00000004), readonly            
]
            IDispatch* CONTROLS;
            [id(0x00000005)            
]
            long top;
            [id(0x00000006)            
]
            long left;
            [id(0x00000007)            
]
            long width;
            [id(0x00000008)            
]
            long height;
            [id(0x00000009), readonly            
]
            VARIANT_BOOL CanFocus;
            [id(0x0000000a), readonly            
]
            VARIANT_BOOL IsVisibleAtRuntime;
            [id(0x0000000b), readonly            
]
            VARIANT_BOOL SingleControl;
            [id(0x0000000c)            
]
            VARIANT_BOOL Visible;
        methods:
            [id(0x0000000d)]
            SCODE Initialize(
                            long hWndControlFrame, 
                            BSTR szName, 
                            IDispatch* pEventsDispatch);
            [id(0x0000000e)]
            IDispatch* AddControl(
                            IDispatch* pControlDef, 
                            long top, 
                            long left, 
                            long width, 
                            long height);
            [id(0x0000000f)]
            void LoadControls(
                            BSTR szFileName, 
                            BSTR szStorageName);
            [id(0x00000010)]
            void SaveControls(
                            BSTR szFileName, 
                            BSTR szStorageName);
            [id(0x00000011)]
            IDispatch* DispatchFromControlName(BSTR lpszCtrlName);
            [id(0x00000012)]
            void ShowHelp();
            [id(0x00000013)]
            IDispatch* ControlAtPoint(
                            long x, 
                            long y);
    };

    [
      uuid(D811D605-0710-4344-8192-DB932AD14194)
    ]
    dispinterface IProControlFrameEvents {
        properties:
        methods:
    };

    [
      uuid(04B58396-9DC6-40C4-A029-2867D58A3946)
    ]
    coclass PROCONTROLFRAME {
        [default] dispinterface IProControlFrame;
        [default, source] dispinterface IProControlFrameEvents;
    };

    [
      uuid(8EF425A1-5D6F-431D-8861-937FAE3EA45A)
    ]
    dispinterface IEvents {
        properties:
            [id(0x00000001), readonly, hidden            
]
            IUnknown* NewEnum;
            [id(0x00000002), readonly            
]
            long Count;
        methods:
            [id(0x00000003)]
            IDispatch* Item(VARIANT Item);
    };

    [
      uuid(14740590-3D9F-4215-B539-BC9C1E54E434)
    ]
    coclass EVENTS {
        [default] dispinterface IEvents;
    };

    [
      uuid(671106D5-3D64-482E-A257-BCF5941FAEA3)
    ]
    dispinterface IProEvent {
        properties:
            [id(0x00000001)            
]
            BSTR Name;
            [id(0x00000002)            
]
            IDispatch* Parameters;
        methods:
    };

    [
      uuid(71B2BF51-DED9-418B-9894-26F32FDA3744)
    ]
    coclass EVENT {
        [default] dispinterface IProEvent;
    };

    [
      uuid(40C3362F-81C6-4076-BB78-2C137887AC2B)
    ]
    dispinterface IParams {
        properties:
            [id(0x00000001), readonly, hidden            
]
            IUnknown* NewEnum;
            [id(0x00000002), readonly            
]
            long Count;
        methods:
            [id(0x00000003)]
            IDispatch* Item(VARIANT Item);
    };

    [
      uuid(DF6CCD1F-0309-4367-B08B-EE0E29BA2B8F)
    ]
    coclass PARAMS {
        [default] dispinterface IParams;
    };

    [
      uuid(A7263339-A87B-47DB-BA65-5A1F506D7CDF)
    ]
    dispinterface IParam {
        properties:
            [id(0x00000001)            
]
            BSTR Name;
            [id(0x00000002)            
]
            BSTR Type;
            [id(0x00000003)            
]
            BSTR Mode;
        methods:
    };

    [
      uuid(0B866C68-B455-4F98-A35B-FC0F819D49BE)
    ]
    coclass PARAM {
        [default] dispinterface IParam;
    };

    [
      uuid(61037F63-F35D-43E9-BE69-C7816E6BFE0A)
    ]
    dispinterface IExtendedObject {
        properties:
            [id(0xfffffce0)            
]
            BSTR Name;
            [id(0x80010001)            
]
            IDispatch* DataSource;
            [id(0x80010002)            
]
            BSTR DataField;
            [id(0x80014444), readonly, hidden            
]
            VARIANT_BOOL _progressprivateIsControl;
            [id(0x80014445)            
]
            BSTR Tag;
            [id(0x80014446)            
]
            VARIANT_BOOL HonorProKeys;
            [id(0x80014447)            
]
            VARIANT_BOOL HonorReturnKey;
            [id(0xfffffcdd), readonly, hidden            
]
            IDispatch* Parent;
            [id(0x80014448)            
]
            VARIANT_BOOL Visible;
        methods:
            [id(0x80014449)]
            SCODE XlateAccelerator(
                            long hWndControl, 
                            long message, 
                            long wParam, 
                            long lParam, 
                            long time, 
                            long x, 
                            long y);
    };

    [
      uuid(DE36CDB2-B05D-4946-9D61-6B569EBC2CC9)
    ]
    dispinterface IExtendedEvents {
        properties:
        methods:
    };

    [
      uuid(87EB6C6C-6EEB-408F-A221-2603BB260FB9)
    ]
    coclass EXTENDEDOBJECT {
        [default] dispinterface IExtendedObject;
        [default, source] dispinterface IExtendedEvents;
    };

    [
      uuid(0340A9DE-4E9B-4245-96FD-986C12E02AD9)
    ]
    dispinterface IProgressRuntime {
        properties:
        methods:
            [id(0x00000001)]
            IDispatch* LoadPicture(BSTR szFileName);
            [id(0x00000002)]
            IDispatch* LoadPictureW(BSTR* pbFileName);
    };

    [
      uuid(EFDC1868-5525-48F3-B5AE-6155A805F102)
    ]
    coclass ProgressRuntime {
        [default] dispinterface IProgressRuntime;
    };
};

Unfortunately it doesn't contain methods for manipulating individual OCX properties. The main method is EditProperties, it handles all the edit flow - UI and persistence.

When looking at the imported system functions of prox.dll the following few look interesting:

StgCreateDocfile
StgOpenStorage
CreateILockBytesOnHGlobal
StgCreateDocfileOnILockBytes
StgOpenStorageOnILockBytes

Apparently it uses COM compound storage for storing the properties (or perhaps only their related data). StgCreateDocfileOnILockBytes creates compound storage on a provided byte buffer, perhaps prox.dll embeds compound storage in wrx files? I.e. the wrx itself is proprietary but part of it is the standard COM compound storage?

Here are some more findings.

Every wrx file is a COM/OLE structured storage file so it can be treated with the COM/OLE Windows system functions like StgOpenStorage.

The structured storage format is a directory like structure. The data is stored in streams (represented by IStream COM interface), streams are placed in directories called storages (represented by IStrogare COM interface). Storages can be nested. Structured storage file can also contain property sets, which always occur at the top level and can't be nested. Property is a name-value structure.

The WRX file contains n number of root storages for every control frame. The storage is named after the control frame's NAME property value. Every control frame storage contains two streams, one named Frame and the other Control0. The WRX file at its root level doesn't contain any property sets.

Now the content of the streams is what's unknown. It is either completely proprietary or it also contains a structured storage blob embedded. I will do some quick tests tomorrow to confirm this.

I created a utility for dumping data of all the streams in a wrx file. I will post this tomorrow.

I have some more findings.

Please note that all the presented results are preliminary as they were tested on a single OCX object. It is possible that different OCX object capabilities (like the supported standard OLE interfaces) may make prox.dll choose different approach how it implements persistence of OCX properties.

When LoadControls is invoked the following happens:
1. prox.dll opens the corresponding wrx and reads the OCX Object IDs.
2. prox.dll uses the Object IDs and loads the type library of the OCXes being loaded, it introspects the OCX and gathers all the design properties, their names, IDs, enum labels and enum values, etc...
3. prox.dll instantiates the OCX object using the IDs loaded from Control0 stream.
4. prox.dll queries IPersistMemory and IPersistStreamInit interfaces from the instantiated OCX. It uses them (or one of them, not yet sure) to load the data from the Control0 stream (or part of it). Hence the property data format we see, if this will be confirmed with further tests, is proprietary to every OCX.
5. prox.dll queries the value of every design property of the OCX previously loaded with the stream data and displays it in the property grid.

We could create a utility that would essentially do the same, but instead of showing the property values in a grid we would output them in a predefined (xml) format in a file and use it during conversion (or runtime if LoadControls is preserved after conversion).

Unknowns:
1. Does prox.dll use all the stream data from the structured storage IStream for deserializing the props or does it use a subset?
2. Does the specific OCX have to support all the OLE COM interfaces mentioned above? How will prox.dll behave when some of the key interfaces are not supported by the OCX? Is this even an issue? Do all OCX's we currently convert support all the above mentioned OLE interfaces?
3. How does prox.dll select the design OCX properties?
4. There seem to be some 4GL specific properties shown in the prox.dll property editor, like Visible or Enabled. If needed these will have to be handled with a different logic.

My next step will be to write the utility to dump the property values and try it on the other OCXes.

I can report a success in parsing OCX properties for PSTimer. I successfully read the Interval and Enabled properties.

But there is still some work required. Especially in deserializing all the various property data types and confirming some of the assumptions (like the offset in the IStream stream where the actual OCX data start).

I will now focus on more complex OCXes and on adding the missing pieces for the proper parser.

That is huge! It will be a big step ahead since we will be able capture the output for all .wrx files up front.

I assume the utility being developed compiles and links using the mingw approach we have in our src/native/. I'm looking forward to seeing the code. What are your thoughts on how to encode the output (e.g. XML)?

Greg Shah wrote:

That is huge! It will be a big step ahead since we will be able capture the output for all .wrx files up front.

I assume the utility being developed compiles and links using the mingw approach we have in our src/native/.

I'm currently compiling the code in Visual Studio, but I plan to convert the project into mingw.

I'm looking forward to seeing the code.

At this point it is very prototypal.

WrxParser.cppWrxParser.cpp

// WrxParser.cpp : This file contains the 'main' function. Program execution begins and ends there.
//

#define WIN32_LEAN_AND_MEAN
#define UNICODE
#define _UNICODE

#include <iostream>
#include <windows.h>
#include <ole2.h>
#include <ocidl.h>

// Implicitly link ole32.dll
#pragma comment( lib, "ole32.lib" )

void printWin32ErrorMessage(DWORD dwErr)
{

    WCHAR   wszMsgBuff[512];  // Buffer for text.

    DWORD   dwChars;  // Number of chars returned.

    // Try to get the message from the system errors.
    dwChars = FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM |
        FORMAT_MESSAGE_IGNORE_INSERTS,
        NULL,
        dwErr,
        0,
        wszMsgBuff,
        512,
        NULL);

    if (0 == dwChars)
    {
        // The error code did not exist in the system errors.
        // Try Ntdsbmsg.dll for the error code.

        HINSTANCE hInst;

        // Load the library.
        hInst = LoadLibrary(L"Ntdsbmsg.dll");
        if (NULL == hInst)
        {
            printf("cannot load Ntdsbmsg.dll\n");
            exit(1);  // Could 'return' instead of 'exit'.
        }

        // Try getting message text from ntdsbmsg.
        dwChars = FormatMessage(FORMAT_MESSAGE_FROM_HMODULE |
            FORMAT_MESSAGE_IGNORE_INSERTS,
            hInst,
            dwErr,
            0,
            wszMsgBuff,
            512,
            NULL);

        // Free the library.
        FreeLibrary(hInst);

    }

    // Display the error message, or generic text if not found.
    printf("Error value: %d Message: %ws\n",
        dwErr,
        dwChars ? wszMsgBuff : L"Error message not found.");

}

class Error 
{
public:

    const wchar_t* msg;
    HRESULT hr;

    Error() 
    {
        msg = L"";
        hr = S_OK;
    }

    Error(const wchar_t* msg, HRESULT hr) 
    {
        this->msg = msg;
        this->hr = hr;
    }

    Error(const Error& copy) 
    {
        msg = copy.msg;
        hr = copy.hr;
    }
};

void checkhr(HRESULT hr, const wchar_t* msg)
{
    if (FAILED(hr))
    {
        throw Error(msg, hr);
    }
}

unsigned char* readControlDataFromWRX(const wchar_t* file)
{
    HRESULT hr = S_OK;
    IStorage* pStorage = NULL;

    hr = StgOpenStorageEx(file,
        STGM_SHARE_EXCLUSIVE | STGM_READ,
        STGFMT_DOCFILE,
        0, NULL, NULL,
        IID_IStorage,
        reinterpret_cast<void**>(&pStorage));

    if (FAILED(hr)) 
    {
        throw Error(L"Failed StgOpenStorageEx", hr);
    }

    pStorage->OpenStorage(L"CtrlFrame", NULL, STGM_SHARE_EXCLUSIVE, NULL, 0, &pStorage);
    if (hr == STG_E_FILENOTFOUND)
    {
        wprintf(L"The root storage doesn't contain storage CtrlFrame, nothing to do.\n");
        return NULL;
    }

    if (FAILED(hr)) 
    {
        throw Error(L"Failed OpenStorage(CtrlFrame)", hr);
    }

    IStream* pStream;
    hr = pStorage->OpenStream(L"Control0", 0, STGM_SHARE_EXCLUSIVE | STGM_READ, 0, &pStream);

    if (hr == STG_E_FILENOTFOUND)
    {
        wprintf(L"The storage doesn't contain stream Control0, nothing to do.\n");
        return NULL;
    }

    if (FAILED(hr)) 
    {
        throw Error(L"Failed OpenStream(Control0)", hr);
    }

    /*
    CoInitialize(NULL);
    IUnknown* pUnknown;
    hr = OleLoadFromStream(pStream, IID_IUnknown, reinterpret_cast<void**>(& pUnknown));
    if (FAILED(hr)) {
        throw Error(L"Failed OleLoadFromStream", hr);
    }
    */

    STATSTG stat;
    hr = pStream->Stat(&stat, STATFLAG_NONAME);
    if (FAILED(hr))
    {
        throw Error(L"Failed stream Stat()", hr);
    }

    size_t size = (size_t)stat.cbSize.QuadPart;
    unsigned char* rawData = (unsigned char*) malloc(size);
    unsigned char* ptr = rawData;
    size_t sizeLeft = size;

    // TODO: read only the CLSID portion
    while (sizeLeft != 0)
    {
        ULONG ul;
        hr = pStream->Read(ptr, (sizeLeft >= MAXDWORD) ? MAXDWORD : (DWORD) sizeLeft, &ul);
        if (FAILED(hr)) 
        {
            throw Error(L"Failed stream Read()", hr);
        }

        if (ul == 0)
        {
            break;
        }
        ptr += ul;
        sizeLeft -= ul;
    }

    // copy the CLSID value from the stream data
    char* clsid = new char[39];
    clsid[38] = 0;
    memcpy(clsid, rawData + 17, sizeof(char) * 38);
    wchar_t* wclsid = new wchar_t[39];
    size_t ret;
    mbstowcs_s(&ret, wclsid, 39, clsid, 38);

    CLSID clsidStr;
    hr = CLSIDFromString(wclsid, &clsidStr);
    if (FAILED(hr))
    {
        throw Error(L"Failed CLSIDFromString", hr);
    }

    // create the target object
    CoInitialize(NULL);
    IUnknown* obj;
    hr = CoCreateInstance(clsidStr, NULL, CLSCTX_INPROC_SERVER, IID_IUnknown, reinterpret_cast<void**>(&obj));
    if (FAILED(hr))
    {
        throw Error(L"Failed CoCreateInstance", hr);
    }

    // TODO: should IPersistMemory be considered, too?
    IPersistStreamInit* persist;
    hr = obj->QueryInterface(IID_IPersistStreamInit, reinterpret_cast<void**>(&persist));
    if (FAILED(hr))
    {
        throw Error(L"Failed QueryInterface(IPersistStream)", hr);
    }

    // TODO: is the serialized object state always at offset 143?
    ULARGE_INTEGER newPos;
    LARGE_INTEGER offset;
    offset.QuadPart = 143;
    pStream->Seek(offset, STREAM_SEEK_SET, &newPos);
    hr = persist->Load(pStream);
    if (FAILED(hr))
    {
        throw Error(L"Failed IPersistStreamInit->Load", hr);
    }

    IDispatch* dispatch;
    hr = obj->QueryInterface(IID_IDispatch, reinterpret_cast<void**>(&dispatch));
    if (FAILED(hr))
    {
        throw Error(L"Failed QueryInterface(IDispatch)", hr);
    }

    ITypeInfo* typeInfo;
    hr = dispatch->GetTypeInfo(0, 0, &typeInfo);
    checkhr(hr, L"dispatch->GetTypeInfo");

    TYPEATTR* pta = nullptr;;
    hr = typeInfo->GetTypeAttr(&pta);
    BSTR bstrName = ::SysAllocString(L"");
    FUNCDESC* pfd = nullptr;
    unsigned int uiNames = 0;

    // TODO: functions won't be needed, remove
    for (int i = 0; i < pta->cFuncs; i++)
    {
        hr = typeInfo->GetFuncDesc(i, &pfd);
        if (FAILED(hr))
        {
            continue;
        }

        hr = typeInfo->GetNames(pfd->memid, &bstrName, 1, &uiNames);
        if (SUCCEEDED(hr) && bstrName && SysStringLen(bstrName))
        {
            wprintf(L"Function: %s\n", bstrName);            
        }
        typeInfo->ReleaseFuncDesc(pfd);
        pfd = nullptr;
    }

    printf("Object properties:\n");

    VARDESC* pvd = nullptr;
    for (int i = 0; i < pta->cVars; i++)
    {
        hr = typeInfo->GetVarDesc(i, &pvd);
        if (FAILED(hr))
        {
            // TODO: log
            continue;
        }

        hr = typeInfo->GetNames(pvd->memid, &bstrName, 1, &uiNames);
        if (FAILED(hr) || !bstrName || !SysStringLen(bstrName))
        {
            // TODO: log
            continue;
        }

        DISPID    dispid;
        hr = dispatch->GetIDsOfNames(
            IID_NULL,
            &bstrName,
            1,
            LOCALE_SYSTEM_DEFAULT,
            &dispid);

        if (FAILED(hr))
        {
            throw Error(L"Failed dispatch->GetIDsOfNames", hr);
        }

        VARIANT value;
        EXCEPINFO exInf;
        DISPPARAMS dispparamsNoArgs = { NULL, NULL, 0, 0 };
        hr = dispatch->Invoke(
            dispid, 
            IID_NULL, 
            LOCALE_SYSTEM_DEFAULT, 
            DISPATCH_PROPERTYGET, 
            &dispparamsNoArgs, 
            &value, 
            &exInf, 
            NULL);

        if (FAILED(hr))
        {
            throw Error(L"Failed dispatch->Invoke", hr);
        }

        if (value.vt == VT_I4)
        {
            printf(": %d\n", value.intVal);
        }
        else
        {
            printf(" (unsupported type %u)\n", value.vt);
        }

        typeInfo->ReleaseVarDesc(pvd);
        pvd = nullptr;
    }

    typeInfo->ReleaseTypeAttr(pta);

    return rawData;
}

EXTERN_C void wmain(int argc, wchar_t* argv[])
{
    if (argc != 2) 
    {
        wprintf(L"Usage: %s <wrx filw>\n", argv[0]);
        return;
    }

    try
    {
        readControlDataFromWRX(argv[1]);
    }
    catch (const Error& e)
    {
        wprintf(L"Error: %s (hr=0x%08x)\n", e.msg, e.hr);
        printWin32ErrorMessage(e.hr);
    }
}

What are your thoughts on how to encode the output (e.g. XML)?

Yes, I think an XML processed during conversion should work. The structure could look like the one below.

<wrx-properties>
  <object id="<an identifier (clsid?)>">
     <property name="" type="" value=""/>
     <property name="" type="" value=""/>
     <property name="" type="" value=""/>
     ...
  </object>
  <object id="<an identifier (clsid?)>">
     <property name="" type="" value=""/>
     <property name="" type="" value=""/>
     <property name="" type="" value=""/>
     ...
  </object>
  ...
</wrx-properties>

It all looks good.

Some of the OCXes require a license key otherwise they fail to instantiate. I found out the license key is part of the wrx file. However I don't know yet how to calculate its offset.

Hynek Cihlar wrote:

Some of the OCXes require a license key otherwise they fail to instantiate. I found out the license key is part of the wrx file. However I don't know yet how to calculate its offset.

Thanks Marius for figuring this out in note #4438-10 :-).

2. The sources must be currently built with Visual Studio until it is ported in MinGW.

Eugenie can help with this.

Greg, where can I commit the sources? You mentioned src/native, but wouldn't tools be a better location? I think it is more in line with things like gen-text-metrics.p.

You aren't wrong that it does fit in tools but there is a benefit to making it easy to build this. If we put it in tools, then I think we should have a makefile. Putting it in src/native/ allows us to leverage that build environment.

Greg Shah wrote:

2. The sources must be currently built with Visual Studio until it is ported in MinGW.

Eugenie can help with this.

Greg, where can I commit the sources? You mentioned src/native, but wouldn't tools be a better location? I think it is more in line with things like gen-text-metrics.p.

You aren't wrong that it does fit in tools but there is a benefit to making it easy to build this. If we put it in tools, then I think we should have a makefile. Putting it in src/native/ allows us to leverage that build environment.

OK I checked the source file in src/native. I will provide the makefile later. See 3821c revision 13540.