Codejock Forums Homepage
Forum Home Forum Home > Codejock Products > Visual C++ MFC > Toolkit Pro
  New Posts New Posts RSS Feed - Crash with Win 10 Version 216.07 and COM
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Crash with Win 10 Version 216.07 and COM

 Post Reply Post Reply
Author
Message
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Topic: Crash with Win 10 Version 216.07 and COM
    Posted: 06 August 2016 at 5:35am
After installing Windows 10 update 1607 and Visual Studio 2015 Update 3 yesterday I have a reproducible (as in 90%) crash in XTP in the destructor of

    ~CXTPComInitializer()
    {
        if(S_OK == m_hr || S_FALSE == m_hr)
        {
            CoUninitialize();
        }
    }

I get an GPF when XTP calls CoUninitialize during the program shut-down. Call stack is

UIAutomationCore.dll!69079d89()    Unknown
[Frames below may be incorrect and/or missing, no symbols loaded for UIAutomationCore.dll]   
UIAutomationCore.dll!6905f64d()    Unknown
combase.dll!NotifyInitializeSpies(int,int,unsigned long,long)    Unknown
combase.dll!_CoUninitialize@0()    Unknown
ToolkitPro1720vc140UD.dll!CXTPComInitializer::~CXTPComInitializer() Line 1286    C++

The m_hr in this case is S_OK which indicates a successful call to CoInitialize in your constructor.

This code is old and the only change was Visual Studio 2015 Upgrade 3 or the most recent Windows update.

I re-compiled XTP after installing the Visual Studio Upgrade.

Back to Top
kstowell View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2003
Location: MIchigan, USA
Status: Offline
Points: 494
Post Options Post Options   Thanks (0) Thanks(0)   Quote kstowell Quote  Post ReplyReply Direct Link To This Post Posted: 27 August 2016 at 9:36am
I am not sure but S_FALSE == m_hr doesn't look correct to me, CoUninitialize() should only be called if a call to CoInitialize() returns S_OK? I know S_FALSE indicates it was already initialized.  I will need to investigate this further to verify, try changing the code to the following, recompile let me know what your results are:

~CXTPComInitializer()
{
    if (S_OK == m_hr/* || S_FALSE == m_hr*/)
    {
        CoUninitialize();
    }
}
Kirk Stowell, President and CEO
CODEJOCK SOFTWARE SOLUTIONS
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 27 August 2016 at 10:59am
The check for S_FALSE is correct, and in accordance with what is stated in MSDN.

CoInitialize returns S_FALSE if the thread has already successfully called CoInitialize. In that case it increments only an internal ref count and returns S_FALSE.

CoUnInitialize must be called for S_OK and S_FALSE. Your code is correct.

The crash does not happen when CoUninitialize is called and m_hr is S_FALSE. This decrements the ref count and works. Only in the last call, when m_hr is S_TRUE, the crash happens. Hence my assumption that there is some 'dangling' COM reference that the system tries to free in that call, and this causes the GPF.

I still have no clue about what's causing this. In my small sampe app with only a main dialog and a second dialog, I can repro the crash100% when I just open and close the second dialog (via a button on the first). But that's just a simple XTPResizableDialog init, no markup is used etc.

As soon as I have a free hour I will try if compiling XTP with
#define _ATL_DEBUG_INTERFACES
#define _ATL_DEBUG_REFCOUNT
reveals leaks.
Back to Top
kstowell View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2003
Location: MIchigan, USA
Status: Offline
Points: 494
Post Options Post Options   Thanks (0) Thanks(0)   Quote kstowell Quote  Post ReplyReply Direct Link To This Post Posted: 27 August 2016 at 11:47am
Ok, it was just a thought, I will have one of our developers take a closer look at this then.  Do you have a set of steps we can follow to recreate the problem?
Kirk Stowell, President and CEO
CODEJOCK SOFTWARE SOLUTIONS
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 29 August 2016 at 9:29am
I'm still trying to figure out what's going on. I try to strip down the test app as much as possible in that process.

I have one main dialog (XTPResizeDialog-derived). On that an XTPButton. When I open the specific dialog via this button and close it with ESC, the crash in ~CXTPComInitializer happens 100% of the time.

So I tried another dialog (About box, also XTPResizeDialog-derived) with the same button. NO crash.

I then stripped down the dialog which causes the crash, removing all code not needed, all message handlers, just binimum OnInitDialog DoDataXchange etc. Whatever I did, the crash still happens when I close my application.

I noticed that the address of the crash various between 0x00000000 and 0x00000028, depending on whether or not I enable XTP skinning for my dialog. Maybe a HINT?

Then another weird thing happened. The crash dialog has controls to open File dialogs. When I open a CFileDialog from the crash dialog, and then close both dialogs and the application, the crash in ~CXTPComInitializer does NOT happen.

Start app > Main dialog > open Crash dialog > close > close app > Crash.
Start app > Main dialog > open Crash dialog > open File Dialog > close > close > close app > NO Crash.

Can this be some kind of hook or something in the XTP code that is triggered? It's always the static variable and it crashes when the XTP DLL shuts down, when in the CXTPMarkupType::CClassList::~CClassList() method.

I had hoped that being able to repro it in a small test app would make it easy to find, but I have still no clue. Maxyb this is a bug in the Microsoft UIAutomationCore.dll shipped with the latest Windows update. Or maybe iit just revealed a dangling problem in XTP...

Still researching







Back to Top
Sven View Drop Down
Senior Member
Senior Member


Joined: 21 August 2003
Location: Germany
Status: Offline
Points: 125
Post Options Post Options   Thanks (0) Thanks(0)   Quote Sven Quote  Post ReplyReply Direct Link To This Post Posted: 29 August 2016 at 10:03am
Are you using COM smart pointers in static objects? In my case (not related to XTP) I had GPFs on exit when a COM smart pointer in a static object was released after CMyWinApp::ExitInstance.
Back to Top
markr View Drop Down
Senior Member
Senior Member


Joined: 01 August 2004
Status: Offline
Points: 391
Post Options Post Options   Thanks (0) Thanks(0)   Quote markr Quote  Post ReplyReply Direct Link To This Post Posted: 29 August 2016 at 10:48am
Originally posted by MacW MacW wrote:

It's always the static variable and it crashes when the XTP DLL shuts down, when in the CXTPMarkupType::CClassList::~CClassList() method.

This is also true for the strange crash reports I've received over the years as well. I've never been able to reproduce them in the development environment, but I get them on a semi-regular basis.

We don't use skinning - appears to be related to the frame hook in our case. Also, we're linking statically.

Recent crashes look like this:

ntdll.dll!RtlpWaitOnCriticalSection() Unknown
  ntdll.dll!RtlpEnterCriticalSectionContended() Unknown
> FolderSizes.exe!CXTPSingletonPointer::GetInstance() Line 68 C++
  FolderSizes.exe!CXTPCommandBarsFrameHook::WinEventProc(CXTPWinEventHook::HWINEVENTHOOK__ * __formal, unsigned long event, HWND__ * hWnd, long idObject, long __formal, unsigned long __formal, unsigned long __formal) Line 114 C++
  [External Code]
  combase.dll!PeekTillDone(HWND__ * hWnd) Line 558 C++
  combase.dll!OXIDEntry::WaitForApartmentShutdown() Line 1543 C++
  combase.dll!OXIDEntry::StopServer() Line 1486 C++
  combase.dll!CComApartment::StopServer() Line 1444 C++
  combase.dll!StopThread(int fHostThread) Line 2315 C++
  combase.dll!ApartmentUninitialize(int fHostThread) Line 2512 C++
  combase.dll!wCoUninitialize(COleTls & Tls, int fHostThread) Line 3894 C++
  combase.dll!CoUninitialize() Line 3814 C++
  FolderSizes.exe!CXTPMarkupObject::~CXTPMarkupObject() Line 627 C++
  [External Code]
  FolderSizes.exe!CCmdTarget::InternalRelease() Line 177 C++
  FolderSizes.exe!CXTPMarkupType::CClassList::~CClassList() Line 294 C++
  FolderSizes.exe!__crt_seh_guarded_call<int>::operator()<<lambda_e24bbb7b643b32fcea6fa61b31d4c984>,<lambda_275893d493268fdec8709772e3fcec0e> & __ptr64,<lambda_9d71df4d7cf3f480f8d633942495c3b0> >(__acrt_lock_and_call::__l3::<lambda_e24bbb7b643b32fcea6fa61b31d4c984> && setup, _execute_onexit_table::__l22::<lambda_275893d493268fdec8709772e3fcec0e> & action, __acrt_lock_and_call::__l4::<lambda_9d71df4d7cf3f480f8d633942495c3b0> && cleanup) Line 199 C++
  FolderSizes.exe!_execute_onexit_table(_onexit_table_t * table) Line 222 C++
  FolderSizes.exe!common_exit(const int return_code, const _crt_exit_cleanup_mode cleanup_mode, const _crt_exit_return_mode return_mode) Line 215 C++
  [External Code]

The fact that you can reproduce your crash scenario consistently is encouraging, at least. Maybe it can help lead us to a solution.

- Mark R.
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 30 August 2016 at 12:18am
Can anybody upload your test application for us to debug or at least if you can add critical changes to any of the sample that we provide and share the patch? We'll try to fix it asap once we have consistent crashing scenario and the code.

Regards,
  Alexander
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2016 at 12:25pm
I'm still trying to figure out what's going on.

Unfortunately it's not easy to strip my test app down. It relies on 9 of my DLLs, a bunch of 3rd party libraries, Open Source frameworks etc...its also multi-threaded. I try to figure out what's causing the "COM leak or whatever"...

I know now that the crash does not happen when I add an extra

OleInitialize(NULL);

into the ExitInstance method of my CWinApp-derived class.

And I have a dialog that causes the behavior 100%, unless I open a CFileDialog from this dialog...when you open a CFileDialog, Windows loads tons of DLLs and whatever and this seems to shuffle something into place which prevents the crash when XTP finally runs its COM cleanup loop.

I really have no idea still.
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2016 at 5:11pm
It can be an issue with late COM uninitialization because of destructors called from static storage. This caused similar issues with other components. However, good news is that in 17.3 we have finally introduced a single point unitialization function called XTPShutdown which is still called during static storage destruction for backward compatibility, but this time it ensures simlutaneous uninitialization of all global data and allows making this call manually for sensitive applications, e.g. in ExitInstance.

Anyway, we still want to pin point all the reported crashes to check their origin and fix whatever is possible.

Regards,
  Alexander
Back to Top
markr View Drop Down
Senior Member
Senior Member


Joined: 01 August 2004
Status: Offline
Points: 391
Post Options Post Options   Thanks (0) Thanks(0)   Quote markr Quote  Post ReplyReply Direct Link To This Post Posted: 06 September 2016 at 3:13pm
MacW, does calling "OleInitialize(NULL)" in ExitInstance working consistently for you in all cases? If so, maybe I'll give it a try and see if we get similar results.

EDIT: It appears that I'm currently calling CoUninitialize in ExitInstance (and CoInitializeEx in InitInstance).

- Mark R.
Back to Top
Marco1 View Drop Down
Senior Member
Senior Member


Joined: 16 January 2004
Location: Germany
Status: Offline
Points: 238
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco1 Quote  Post ReplyReply Direct Link To This Post Posted: 06 September 2016 at 4:53pm
FYI, this is how my ExitInstance looks the last years. It didn't change and is running fine:

int CApp::ExitInstance()
{
  // release ole message filters
  COleMessageFilter* pFilter = AfxOleGetMessageFilter();
  ULONG lCount;
  while ((lCount = pFilter->m_xMessageFilter.Release()) > 2);
  if (lCount <= 1) {
    pFilter->m_xMessageFilter.AddRef();
  }

  // Close the COM library
  CoUninitialize();

  Gdiplus::GdiplusShutdown(m_gdiplusToken);

  // Stop skin
  XTPSkinManager()->LoadSkin(NULL, NULL);
  XTPSkinManager()->RemoveAll();

  CXTPPaintManager::Done();
  XTPImageManager()->RemoveAll();

  XTPResourceManager()->Close();

  return CWinApp::ExitInstance();
}


Product: XTP 18.3.0 on VS 2017
Platform: VS 2017 / Windows 10 (64bit)
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 06 September 2016 at 9:51pm
I would recommend placing CoUninitialize and GdiplusShutdown calls after cleaning up all XTP resources, right before ExitInstance, as XTP can still hold COM references that will be released only after COM uninitialization in your case, which can lead to unpredicted behavior. Releasing OLE message filter in such a way look more like a hack to me and is error prone, it's better to invest time in making sure all references are released prior to ExitInstance in a natural way than deal with consequences of extra release calls.
Regards,
  Alexander
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 07 September 2016 at 9:07am
Originally posted by markr markr wrote:

MacW, does calling "OleInitialize(NULL)" in ExitInstance working consistently for you in all cases? If so, maybe I'll give it a try and see if we get similar results.

EDIT: It appears that I'm currently calling CoUninitialize in ExitInstance (and CoInitializeEx in InitInstance).

- Mark R.


I meant adding an extra CoInitialize(NULL) (init not de-init) right before calling ExitInstance of the super class prevented the crash in my both test cases.


...
CoInitialize(NULL);

return __super::ExitInstance();

In this case the static destructor in XTP runs through as usual but does not no crash.
My application shuts down properly, no crashes or anything.
I just don't know if this causes any side-effects.

I don't call any other XTP cleanup routines.
When I add the cleanup stuff you do, there is no difference. The crash still happens without the extra call to CoInitialize.
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 10 September 2016 at 4:34pm
This only indicates that there are some dangling COM references which get released after ExitInstance. I cannot know who holds those references as I have no sample that has this issue, it can be both a bug in XTP or the application that uses it. Anywayt with new XTPShutdown function it will be possible to make sure XTP cleanup occurs before COM uninitialization if you call it explicitly.

Regards,
  Alexander
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 18 October 2016 at 8:36am
After installing Service Pack  3 of Visual Studio, my trick with the extra OleInitialize in Exitinstance no longer works. XTP crashes during shut-down reliably in

CXTPMarkupObject::~CXTPMarkupObject()
...
SAFE_DELETE(m_pComInitializer);  <<

Call stack is

     ntdll.dll!7760e8af()    Unknown
     ntdll.dll!7761ff1f()    Unknown
     combase.dll!74321510()    Unknown
     combase.dll!742e2d0c()    Unknown
     combase.dll!742e29d0()    Unknown
     ToolkitPro1720vc140U.dll!CXTPMarkupObject::~CXTPMarkupObject() Line 627    C++
     ToolkitPro1720vc140U.dll!CXTPMarkupType::~CXTPMarkupType() Line 390    C++
     ToolkitPro1720vc140U.dll!CXTPMarkupType::`vector deleting destructor'(unsigned int)    C++
>    ToolkitPro1720vc140U.dll!CXTPMarkupObject::OnFinalRelease() Line 643    C++
     mfc140u.dll!01f44610()    Unknown
     ToolkitPro1720vc140U.dll!CXTPMarkupType::CClassList::~CClassList() Line 294    C++
     ucrtbase.dll!74e23489()    Unknown
     ucrtbase.dll!74e2057b()    Unknown
     ucrtbase.dll!74e2325e()    Unknown
     ToolkitPro1720vc140U.dll!__scrt_dllmain_uninitialize_c() Line 395    C++
     ToolkitPro1720vc140U.dll!dllmain_crt_process_detach(const bool is_terminating) Line 109    C++
     ToolkitPro1720vc140U.dll!dllmain_crt_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 134    C++
     ToolkitPro1720vc140U.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 207    C++
     ToolkitPro1720vc140U.dll!_DllMainCRTStartup(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 248    C++

Back to Top
cpede View Drop Down
Senior Member
Senior Member


Joined: 13 August 2004
Status: Offline
Points: 541
Post Options Post Options   Thanks (0) Thanks(0)   Quote cpede Quote  Post ReplyReply Direct Link To This Post Posted: 19 October 2016 at 6:55pm
I have the exact same problem.

If I open a dialog in my program and before exiting, I get the exception.

-cpede
Product: Xtreme ToolkitPro (18.0.1)
Platform: Windows 10 (x64)
Language: Visual Studio 2017 (C++)
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (1) Thanks(1)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 19 October 2016 at 10:36pm
This issue has been fixed for 17.3. The update is about to be released. Once it's out please re-check it and let me know if it still happens.

Regards,
  Alexander
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 20 October 2016 at 8:45am
This sounds great. I test it as soon as your new release comes out.
Back to Top
TomPP View Drop Down
Newbie
Newbie


Joined: 17 November 2014
Location: Germany
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote TomPP Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2016 at 11:42am
Hi,

still crashes for me in i.e. the GridSample of 17.3 on Windows 10 pro 1607 and VC 2015 SP3.
Just starting and stopping the sample throws an exception in the CoUninitializelLine.

    ~CXTPComInitializer()
    {
        if(S_OK == m_hr || S_FALSE == m_hr)
        {
            CoUninitialize();
        }
    }

Next try was the ReportDialog Sample. After exiting the Sample (just start and exit)
it shows a lot of memory leaks.

ToolkitPro.cpp(59) : atlTraceGeneral - Codejock ToolkitPro Terminating!
Detected memory leaks!
Dumping objects ->
..\..\Source\ReportControl\XTPReportControl.cpp(5894) : {1744} normal block at 0x00C48500, 160 bytes long.
 Data: <                > 90 FE 8C 02 02 00 00 00 00 00 00 00 00 00 00 00
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\strcore.cpp(156) : {928} normal block at 0x00C58A18, 18 bytes long.
 Data: <                > 80 03 0D 10 01 00 00 00 01 00 00 00 01 00 00 00
c:\source\vc15 libs\xtreme toolkitpro v17.3.0\samples\reportcontrol\reportdialog\messagerecord.cpp(250) : {927} normal block at 0x00C80C48, 176 bytes long.
 Data: <L               > 4C 17 B7 00 01 00 00 00 00 00 00 00 00 00 00 00
..\..\Source\ReportControl\XTPReportRecordItem.cpp(1779) : {924} normal block at 0x00C7F2C8, 52 bytes long.
...
...
...
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\dumpcont.cpp(23) : atlTraceGeneral - a CCmdTarget object at $00C31F48, 224 bytes long
Object dump complete.
Das Programm "[9336] ReportDialog.exe" wurde mit Code 0 (0x0) beendet.

A simple XTPReportControls in a Dialog or PropertyPage chrashes on exit too.

No problems so far in Windows 7 pro.

If you need more input, please let me know.

Tom
Back to Top
MacW View Drop Down
Senior Member
Senior Member


Joined: 26 June 2007
Status: Offline
Points: 250
Post Options Post Options   Thanks (0) Thanks(0)   Quote MacW Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2016 at 4:10am
17.3 still crashes in the call to

SAFE_DELETE(m_pComInitializer);

in the file XTPMarkupObject.cpp.

I have added a call to XTPShutdown() right before I call CWinApp::ExitInstance() in my derived class.
The crash happens during the shut-down code of MFC, my code is already cleaned up at the time.

it's the static variables XTP uses that are causing this dreadful problem. We need to get this fixed asap.
We waited so long for the 17.3 but the problem still exists in your code.

Please fix or post a work-around in days, not months. Thank you.
I don't care for anything else in XTP unless this reproducible crash is fixed.
Back to Top
Felipe View Drop Down
Groupie
Groupie


Joined: 08 August 2003
Status: Offline
Points: 12
Post Options Post Options   Thanks (0) Thanks(0)   Quote Felipe Quote  Post ReplyReply Direct Link To This Post Posted: 04 January 2017 at 10:12am
Same crash here with Windows 10 and v17.3

Please publish a solution or a workaround, we are blocked.
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 05 January 2017 at 12:21am
We're currently working on a fix, a patch will be provided within a few days.

Regards,
  Alexander
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 12 January 2017 at 7:37pm
The issue has been fixed. The patch for ToolkitPro 17.3 can be downloaded from here - uploads/8199/TID-23088.zip (unidiff log + modified sources included). Please let me know if it works properly or not.

Regards,
  Alexander
Back to Top
TomPP View Drop Down
Newbie
Newbie


Joined: 17 November 2014
Location: Germany
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote TomPP Quote  Post ReplyReply Direct Link To This Post Posted: 12 January 2017 at 11:09pm
Hi,

this patch did not fix the problem.

Toolkit pro 17.3, Windows 10 pro 1607, VC 2015 SP3

Applied the patch and rebuilded the library...

Compiled and started your GridSample...

On exit..

Crashes somtimes with this exception (translated from the german output):

Thread 0x1b44 has ended with code 0.
Raised exception: read access violation
"this" was "0xDDDDDE05".

Line 271 in afxtempl.h

...
/*============================================================================*/
// CArray<TYPE, ARG_TYPE> inline functions

template<class TYPE, class ARG_TYPE>
AFX_INLINE INT_PTR CArray<TYPE, ARG_TYPE>::GetSize() const
---here-->>>    { return m_nSize; }
...


Callstack:
>    ToolkitPro1730vc140D.dll!CArray<CXTPReportColumn *,CXTPReportColumn *>::GetSize() Zeile 271    C++
     ToolkitPro1730vc140D.dll!CXTPReportColumns::GetCount() Zeile 349    C++
     ToolkitPro1730vc140D.dll!CXTPReportColumns::Find(int nItemIndex) Zeile 258    C++
     ToolkitPro1730vc140D.dll!CXTPReportRow::GetAccessibleState(tagVARIANT varChild, tagVARIANT * pvarState) Zeile 1788    C++
     ToolkitPro1730vc140D.dll!CXTPAccessible::XAccessible::get_accState(tagVARIANT varChild, tagVARIANT * pvarState) Zeile 888    C++
     [Externer Code]    
     [Unten angegebene Rahmen sind möglicherweise nicht korrekt und/oder fehlen, keine Symbole geladen für oleacc.dll]    
     ToolkitPro1730vc140D.dll!CXTPControl::OnExecute() Zeile 573    C++
     [Externer Code]    
     GridSample.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Zeile 26    C++
     [Externer Code]    


And way more often crashes this way:

Thread 0x1140 ended with code 0.
..\..\Source\Common\XTPApplication.cpp(72) : atlTraceGeneral - Codejock XTP application shutting down ...
Exception raised at 0x69BDA0C9 (UIAutomationCore.dll) in GridSample.exe: 0xC0000005: Read access violation at position 0x00000008.

In file XTPSystemHelpers.h line 1289
...
    ~CXTPComInitializer()
    {
        if(S_OK == m_hr || S_FALSE == m_hr)
        {
---here--->            CoUninitialize();
        }
    }
...

Callstack:
     UIAutomationCore.dll!69bda0c9()    Unbekannt
     [Unten angegebene Rahmen sind möglicherweise nicht korrekt und/oder fehlen, keine Symbole geladen für UIAutomationCore.dll]    
     [Externer Code]    
>    ToolkitPro1730vc140D.dll!CXTPComInitializer::~CXTPComInitializer() Zeile 1289    C++
     [Externer Code]    
     ToolkitPro1730vc140D.dll!CXTPMarkupObject::~CXTPMarkupObject() Zeile 626    C++
     ToolkitPro1730vc140D.dll!CXTPMarkupType::~CXTPMarkupType() Zeile 389    C++
     [Externer Code]    
     ToolkitPro1730vc140D.dll!CXTPMarkupObject::OnFinalRelease() Zeile 642    C++
     [Externer Code]    
     ToolkitPro1730vc140D.dll!CXTPMarkupObject::Release() Zeile 648    C++
     ToolkitPro1730vc140D.dll!CXTPMarkupType::CClassList::~CClassList() Zeile 294    C++
     ToolkitPro1730vc140D.dll!CXTPSingleton<CXTPMarkupType::CClassList>::Destroy(void * ptr) Zeile 112    C++
     ToolkitPro1730vc140D.dll!CXTPSingletonPointer::Destroy() Zeile 105    C++
     ToolkitPro1730vc140D.dll!CXTPSingletonPointer::OnApplicationShutdown(CXTPApplication * pApplication) Zeile 123    C++
     ToolkitPro1730vc140D.dll!CXTPObservable<CXTPApplication,IXTPApplicationEvents>::Notify(void(IXTPApplicationEvents::*)(CXTPApplication *) method) Zeile 1395    C++
     ToolkitPro1730vc140D.dll!CXTPApplication::Shutdown() Zeile 75    C++
     ToolkitPro1730vc140D.dll!XTPShutdown() Zeile 93    C++
     ToolkitPro1730vc140D.dll!DllMain(HINSTANCE__ * hInstance, unsigned long dwReason, void * __formal) Zeile 59    C++
     [Externer Code]    

Please keep working on the problem as this release is unusable in the mentioned configuration.

Tom
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 13 January 2017 at 12:03am
Thank you, Tom, for reporting this quickly. I haven't tested with this particular sample and was able to reproduce the issue right away, it's a good indication it'll be fixed soon, it's just something minor that has been overlooked.

Regards,
  Alexander
Back to Top
Felipe View Drop Down
Groupie
Groupie


Joined: 08 August 2003
Status: Offline
Points: 12
Post Options Post Options   Thanks (0) Thanks(0)   Quote Felipe Quote  Post ReplyReply Direct Link To This Post Posted: 13 January 2017 at 7:01am
Hi I tested the patch here and still have the crash.
In my case however I had solved the problem by adding  CoInitialize(NULL) in the application ExitInstance().
With the patch I still need CoInitialize(NULL) to avoid crash.

Cheers
Felipe
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 13 January 2017 at 8:18pm
I appreciate everyone's help with debugging the issue. Please try out a new patch with affected source files (previous fix included) - uploads/8199/TID-23088-2.zip

Regards,
  Alexander

EDITED: Do not use this patch as it contains a bug. Read my next message for details.
Back to Top
TomPP View Drop Down
Newbie
Newbie


Joined: 17 November 2014
Location: Germany
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote TomPP Quote  Post ReplyReply Direct Link To This Post Posted: 14 January 2017 at 1:51am
Hi,

i tested the new patch with the GridSample and it throws the same exceptions.

So i inspected the code and my findings are:

The CoUninitialize() exception only raises when using ToolkitPro as a DLL. When using a static build
everything works fine so far regarding the CoUninitialize() problem.

You are calling CoInitialize() and CoUninitialize() in the DllMain. Microsoft says: "Because there is no
way to control the order in which in-process servers are loaded or unloaded, do not call CoInitialize,
CoInitializeEx, or CoUninitialize from the DllMain function."

CoInitialize() should be called for every thread. Is this guaranteed when it is only called during the
construction of the markup objects and the singleton approach?

Even in a static build i still get the following error i mentioned in my last post. But i cant reproduce
it as it raises at about every tenth run:

Thread 0x1b44 has ended with code 0.
Raised exception: read access violation
"this" was "0xDDDDDE05".

Line 271 in afxtempl.h

...
/*============================================================================*/
// CArray<TYPE, ARG_TYPE> inline functions

template<class TYPE, class ARG_TYPE>
AFX_INLINE INT_PTR CArray<TYPE, ARG_TYPE>::GetSize() const
---here-->>>    { return m_nSize; }
...


Callstack:
>    ToolkitPro1730vc140D.dll!CArray<CXTPReportColumn *,CXTPReportColumn *>::GetSize() Zeile 271    C++
     ToolkitPro1730vc140D.dll!CXTPReportColumns::GetCount() Zeile 349    C++
     ToolkitPro1730vc140D.dll!CXTPReportColumns::Find(int nItemIndex) Zeile 258    C++
     ToolkitPro1730vc140D.dll!CXTPReportRow::GetAccessibleState(tagVARIANT varChild, tagVARIANT * pvarState) Zeile 1788    C++
     ToolkitPro1730vc140D.dll!CXTPAccessible::XAccessible::get_accState(tagVARIANT varChild, tagVARIANT * pvarState) Zeile 888    C++
     [Externer Code]    
     [Unten angegebene Rahmen sind möglicherweise nicht korrekt und/oder fehlen, keine Symbole geladen für oleacc.dll]    
     ToolkitPro1730vc140D.dll!CXTPControl::OnExecute() Zeile 573    C++
     [Externer Code]    
     GridSample.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Zeile 26    C++
     [Externer Code]    

Hope this helps.

Tom
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 14 January 2017 at 10:15pm
Hi Tom,

You are absolutely right about COM initialization requirements in multi-threaded application. The second patch did not take this requirement into account so it is not correct, I have reverted it in the source code and will ask you and others to use only the first patch - uploads/8199/TID-23088.zip.

You are also right about making a call to CoUninitialize from DllMain, however starting from 17.3 you can now control this by calling XTPShutdown from a proper place.

As for the crashes that you are still getting, what I think this time it's not because of the bug in the code, it's rather because of the UI Automation library that you're using (I figured it out from the call-stack).

What happens is when you start GridSample it does not initialize COM itself, instead some internal XTP classes initialize COM for their own purposes, thus when the application is shutting down XTP classes uninitialize COM which unloads COM libraries completely because there are no other references to COM. However your UI Automation libraries still use COM, because it's how Automation is implemented. So, with COM libraries unloaded the first Automation call leads to undefined behavior, or a crash in this case. Adding CoInitialize call to CGridSampleApp constructor WITHOUT adding CoUnitialize call in its destructor will prevent from COM unloading and fix your crash.

Also if you run GridSample without UI Automation attached you won't see the crash.

Please let me know if you are able to confirm my theory.

Regards,
  Alexander




Back to Top
sbinder View Drop Down
Groupie
Groupie
Avatar

Joined: 22 November 2004
Location: Austria
Status: Offline
Points: 74
Post Options Post Options   Thanks (0) Thanks(0)   Quote sbinder Quote  Post ReplyReply Direct Link To This Post Posted: 20 January 2017 at 8:04am
If I call XTPShutdown() in ExitInstance before CWinApp::ExitInstance() is called
my application crashs when Windows is trying to (my application prevents it) shut down (Win7, 8.1 and 10)!

Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2017 at 12:36am
Sbinder, can you please provide a stack trace for the crash? Also please make sure all COM references are released properly in your scenario prior to XTPShutdown call. It is recommended to call it as late as possible.

Regards,
  Alexander
Back to Top
sbinder View Drop Down
Groupie
Groupie
Avatar

Joined: 22 November 2004
Location: Austria
Status: Offline
Points: 74
Post Options Post Options   Thanks (0) Thanks(0)   Quote sbinder Quote  Post ReplyReply Direct Link To This Post Posted: 23 January 2017 at 6:04am

Call Stack:

  XTP0173xd.dll!CXTPSingletonPointer::GetInstance()  Line 60 + 0x19 bytes C++
  XTP0173xd.dll!CXTPSingleton<CXTPHookManager>::Instance()  Line 123 + 0xc bytes C++
> XTP0173xd.dll!XTPHookManager()  Line 349 C++
  XTP0173xd.dll!CXTPHookManager::HookWndProc(HWND__ * hWnd=0x00111052, unsigned int message=22, unsigned int wParam=1, long lParam=0)  Line 435 + 0x9 bytes C++
  user32.dll!__InternalCallWinProc@20()  + 0x2b bytes 
  user32.dll!UserCallWinProcCheckWow()  + 0x30a bytes 
  user32.dll!DispatchClientMessage()  + 0xf0 bytes 
  user32.dll!___fnDWORD@4()  + 0x49 bytes 
  ntdll.dll!_KiUserCallbackDispatcher@12()  + 0x36 bytes 
  win32u.dll!_NtUserMessageCall@28()  + 0xc bytes 
  user32.dll!RealDefWindowProcWorker()  + 0xba bytes 
  user32.dll!RealDefWindowProcA()  + 0x5d bytes 
  mfc100d.dll!AfxUnlockGlobals()  + 0xb4 bytes 
  user32.dll!__InternalCallWinProc@20()  + 0x2b bytes 
  user32.dll!UserCallWinProcCheckWow()  + 0x30a bytes 
  user32.dll!CallWindowProcA()  + 0x98 bytes 
  mfc100d.dll!CWnd::DefWindowProcA()  + 0x34 bytes 
  mfc100d.dll!CWnd::WindowProc()  + 0x52 bytes 
  mfc100d.dll!AfxCallWndProc()  + 0xf3 bytes 
  mfc100d.dll!AfxWndProc()  + 0xa6 bytes 
  mfc100d.dll!AfxWndProcBase()  + 0x5b bytes 
  user32.dll!__InternalCallWinProc@20()  + 0x2b bytes 
  user32.dll!UserCallWinProcCheckWow()  + 0x30a bytes 
  user32.dll!CallWindowProcA()  + 0x98 bytes 
  XTP0173xd.dll!CXTPHookManager::HookWndProc(HWND__ * hWnd=0x00111052, unsigned int message=59, unsigned int wParam=268, long lParam=0)  Line 431 + 0x1c bytes C++
  user32.dll!__InternalCallWinProc@20()  + 0x2b bytes 
  user32.dll!UserCallWinProcCheckWow()  + 0x30a bytes 
  user32.dll!DispatchClientMessage()  + 0xf0 bytes 
  user32.dll!___fnDWORD@4()  + 0x49 bytes 
  ntdll.dll!_KiUserCallbackDispatcher@12()  + 0x36 bytes 
  user32.dll!GetMessageA()  + 0x53 bytes 
  mfc100d.dll!AfxInternalPumpMessage()  + 0x23 bytes 
  mfc100d.dll!CWinThread::PumpMessage()  + 0xe bytes 
  mfc100d.dll!CWinThread::Run()  + 0x8d bytes 
  mfc100d.dll!CWinApp::Run()  + 0x59 bytes 
  mfc100d.dll!AfxWinMain()  + 0xef bytes 
  LogoMed.exe!WinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * hPrevInstance=0x00000000, char * lpCmdLine=0x01a25766, int nCmdShow=1)  Line 26 C++
  LogoMed.exe!__tmainCRTStartup()  Line 547 + 0x2c bytes C
  LogoMed.exe!WinMainCRTStartup()  Line 371 C
  kernel32.dll!@BaseThreadInitThunk@12()  + 0x24 bytes 
  ntdll.dll!__RtlUserThreadStart()  + 0x2f bytes 
  ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes 

Back to Top
sbinder View Drop Down
Groupie
Groupie
Avatar

Joined: 22 November 2004
Location: Austria
Status: Offline
Points: 74
Post Options Post Options   Thanks (0) Thanks(0)   Quote sbinder Quote  Post ReplyReply Direct Link To This Post Posted: 23 January 2017 at 6:05am
I call XTPShudown() before CWinApp::ExitInstance(), this is as late as possible. I can't find a unreleased COM-Reference.

I get a Debug Assert in XTPSingleton.cpp Line 60:

void* CXTPSingletonPointer::GetInstance()
{
 ASSERT(!m_bDestroyed);
...
Back to Top
astoyan View Drop Down
Admin Group
Admin Group
Avatar

Joined: 24 August 2013
Status: Offline
Points: 184
Post Options Post Options   Thanks (0) Thanks(0)   Quote astoyan Quote  Post ReplyReply Direct Link To This Post Posted: 25 January 2017 at 11:27pm
I cannot see a XTPShutdown on your stack trace so I assume the error occurs before or after it is called. Are you using SkinFramework? If so, have you tried to unload skin before making a call to XTPShutdown?
The best option in this case would be a working sample so that I can see exactly what and how you're doing instead of making and testing guesses. Is it possible for you to make it or modify any existing one?

Regards,
  Alexander
Back to Top
sbinder View Drop Down
Groupie
Groupie
Avatar

Joined: 22 November 2004
Location: Austria
Status: Offline
Points: 74
Post Options Post Options   Thanks (0) Thanks(0)   Quote sbinder Quote  Post ReplyReply Direct Link To This Post Posted: 03 February 2017 at 8:05am
I am not using SkinFramework, but I use Mapi and I assume that the problem is related to it. I am currenty missing the time to create an example.
Back to Top
Don View Drop Down
Newbie
Newbie


Joined: 07 March 2017
Status: Offline
Points: 1
Post Options Post Options   Thanks (0) Thanks(0)   Quote Don Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2017 at 3:03pm
I recently started to see a crash in my app just like the one described in the original post. I have a Windows 10 laptop that does not have a touch screen. I believe the crash started happening when I installed an app that lets you use an iPad as a second monitor for your PC (duetdisplay). That app seems to install touch input drivers which lets it accept touch screen input from the iPad. I believe that the touch input drivers are the reason for UIAutomationCore.dll appearing in the call stack, and are a factor in causing this crash.

I haven't tried the patch posted above, but I thought I'd give you this info in case it would help reproduce the problem.
Back to Top
rdhd View Drop Down
Senior Member
Senior Member
Avatar

Joined: 13 August 2007
Location: United States
Status: Offline
Points: 607
Post Options Post Options   Thanks (0) Thanks(0)   Quote rdhd Quote  Post ReplyReply Direct Link To This Post Posted: 23 May 2018 at 12:23pm
We have CJ 18.2. And I am getting this same crash issue with the crash in UIAutomation. I have to have an actual ribbon created. So apparently this hasn't been fixed? Any progress on it?

     UIAutomationCore.dll!std::_Hash<std::_Umap_traits<unsigned long,CApartmentTracker * __ptr64,std::_Uhash_compare<unsigned long,std::hash<unsigned long>,std::equal_to<unsigned long> >,std::allocator<std::pair<unsigned long const ,CApartmentTracker * __ptr64> >,0> >::lower_bound()    Unknown
     UIAutomationCore.dll!CApartmentTracker::PreUninitialize(unsigned long)    Unknown
     combase.dll!NotifyInitializeSpies(int fInitialize, int fPreNotify, unsigned long dwFlags, HRESULT hrInit) Line 3151    C++
     combase.dll!CoUninitialize() Line 3903    C++
>    ToolkitPro1820vc140x64U.dll!CXTPMarkupObject::~CXTPMarkupObject() Line 624    C++
     ToolkitPro1820vc140x64U.dll!CXTPMarkupType::`vector deleting destructor'(unsigned int)    C++
     mfc140u.dll!CCmdTarget::InternalRelease() Line 177    C++
     ToolkitPro1820vc140x64U.dll!CXTPMarkupType::CClassList::~CClassList() Line 294    C++
     ToolkitPro1820vc140x64U.dll!CXTPSingletonPointer::Destroy() Line 120    C++
     ToolkitPro1820vc140x64U.dll!CXTPApplication::Shutdown() Line 81    C++
     ToolkitPro1820vc140x64U.dll!XTPShutdown() Line 106    C++
     ToolkitPro1820vc140x64U.dll!DllMain(HINSTANCE__ * hInstance, unsigned long dwReason, void * __formal) Line 65    C++
     ToolkitPro1820vc140x64U.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 195    C++
     ntdll.dll!LdrpCallInitRoutine()    Unknown
     ntdll.dll!LdrShutdownProcess()    Unknown
     ntdll.dll!RtlExitUserProcess()    Unknown
     kernel32.dll!ExitProcessImplementation()    Unknown
     ucrtbase.dll!exit_or_terminate_process()    Unknown
     ucrtbase.dll!common_exit()    Unknown
     Edge.exe!__scrt_common_main_seh() Line 262    C++

Back to Top
rdhd View Drop Down
Senior Member
Senior Member
Avatar

Joined: 13 August 2007
Location: United States
Status: Offline
Points: 607
Post Options Post Options   Thanks (0) Thanks(0)   Quote rdhd Quote  Post ReplyReply Direct Link To This Post Posted: 23 May 2018 at 5:09pm
Notice that the call to Couninitialize is via DllMain. The MSDN says this:

Because there is no way to control the order in which in-process servers are loaded or unloaded, do not call CoInitialize, CoInitializeEx, or CoUninitialize from the DllMain function.

In my case, the crash doesn't always occur. Could this be a timing issue? The only thread left when I crash is the main thread. I also happen to have a desktop box and one of my monitors (the one the app is on) is a touch screen. I mention that after seeing the reply about using an iPad.

We have always called CoInitialize soon after we start up. But it is not during the call to DllMain. I did just add a call to CoInitialize in another class constructor that gets called before the first call CJ makes. That keeps CJ from crashing during shutdown. But that just moved the crash to the CoUninitialze call I added at the same time in the destructor. But that code is called during our DllMain too. Same type of setup CJ has - we have a static object that gets created/destroyed during the dll init phase. Hence, I violated the MSDN rule on calling from DllMain. I only chose it because it was the only way I had to beat CJ and be the first to call CoInitialize.

Since CJ is doing this during DLL loading, I really have no other way to beat CJ to the call. I can keep the crash from occurring by not calling the CoUnitialize call I added to compensate for the call to init I added.

My best guess is that CJ needs to move this out of the DllMain code path. Why are these markup objects needed during process load time?

UIAutomationCore is trying to uninit something called CApartmentTracker. There is a std::map of long to CApartmentTracker*. But I can't tell what it is doing other than trying to load data from an address that is invalid (causes the AV). I have found no documentation on this CApartmentTracker class. But the class code is in uiautomationcore.dll. Is it tracking something in a DLL that has been unloaded?
Back to Top
TomPP View Drop Down
Newbie
Newbie


Joined: 17 November 2014
Location: Germany
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote TomPP Quote  Post ReplyReply Direct Link To This Post Posted: 24 May 2018 at 5:13am
What i found out....

I use a password manager (Sticky Password) to manage my accounts. This manager has an option
to integrate via UI Automation. When enabled, this COM error occurs within programs using the
CJ Toolkit but in no other software. When disabled, no error occurs at all.

This belongs only to Windows 10, with Win 7 everything is fine.

Tom 
Back to Top
rdhd View Drop Down
Senior Member
Senior Member
Avatar

Joined: 13 August 2007
Location: United States
Status: Offline
Points: 607
Post Options Post Options   Thanks (0) Thanks(0)   Quote rdhd Quote  Post ReplyReply Direct Link To This Post Posted: 24 May 2018 at 12:13pm
I have modified our application shutdown code run after we exit the process messages loop. We call our own shutdown APIs. I added a call to the CJ shutdown function. So far, that keeps the crash from occurring.

I also noticed my (new) box has Win 10 version 1607 on it. I tested another box we use in certification because I should have heard about this issue from testing. No crashes there using my same workflow. But that box has version 1709 on it. I'm updating my box to 1709 after which I will undo my change and see if the crash can be reproduced. Perhaps this is an OS issue that has been fixed.
Back to Top
rdhd View Drop Down
Senior Member
Senior Member
Avatar

Joined: 13 August 2007
Location: United States
Status: Offline
Points: 607
Post Options Post Options   Thanks (0) Thanks(0)   Quote rdhd Quote  Post ReplyReply Direct Link To This Post Posted: 24 May 2018 at 3:55pm
I had the OS update to 1709. The crash goes away. I have not tried 1803 (our company won't let us have it yet).

I suspect some DLL unloading order was changed in the update I applied. So, I plan to still call XTPShutdown() from our shutdown code to prevent CoUninitialize from being called from CJ's DllMain since that fixes the issue too. That way any OS or other change that might affect the DLL unloading order won't cause this to show back up.
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.125 seconds.