Ticket #4204 (closed: fixed)
Ensure old- and new-style Python APIs can coexist in the same process
| Reported by: | Martyn Gigg | Owned by: | Martyn Gigg |
|---|---|---|---|
| Priority: | blocker | Milestone: | Release 2.0 |
| Component: | Mantid | Keywords: | |
| Cc: | Blocked By: | ||
| Blocking: | Tester: | Nick Draper |
Description
At the moment if you try and import the new mantid package and the old MantidFramework in to the same process you get a boost assertion error that some registrations already exist and the program exits.
The issue is that boost::python's type conversion registry is a static map and is shared among all dynamically loaded extension modules. The solution is to statically link the old API to boost python so that it gets a private copy of the registry and they don't clash.
We don't want to statically link the new one however as we have split modules that need to share the registry.
Change History
comment:2 Changed 9 years ago by Martyn Gigg
Refs #4204. Make the old/new python interfaces will blissfully coexist.
The ticket is the best place to look for the reason why this is so convoluted. Please also read 'PythonAPI/src/why_is_boost_python_here.txt'.
Changeset: be5e8879809bc64fc6ff1799e16080d4c5aeb915
comment:3 Changed 9 years ago by Martyn Gigg
Refs #4204. Fix for RHEL5 build and kill some warnings.
Changeset: 946b49ab6b55c4026d3920a371e899f4cae58dc9
comment:4 Changed 9 years ago by Martyn Gigg
Refs #4204. Fix for the builds working off Third_Party.
Changeset: 1664665b1359e126f8972d2fbcda49a93285392c
comment:5 Changed 9 years ago by Martyn Gigg
Refs #4204. RHEL6 needs to load stdc++ very early.
If stdc++ is not loaded early the thrown exceptions cause crashes instead.
Changeset: 4bf9327ed4c73651083743167ae0a77d99734620
comment:6 Changed 9 years ago by Martyn Gigg
Refs #4204. Fix for running from a directory outside bin.
Changeset: e3eb756967bc4659230a2ff38177645de4e88bce
comment:7 Changed 9 years ago by Martyn Gigg
Refs #4204. Change library lookup to one that is always in bin.
The old Python framewoek was looking at the depenencies on Kernel but this is not always next to the MantidFramework file. Move to check against libMantidPythonAPI that will always be next to the file.
Changeset: 5d006b76452f88e20852047ea86cbffbe0264f47
comment:9 Changed 9 years ago by Martyn Gigg
- Status changed from accepted to verify
- Resolution set to fixed
comment:10 Changed 9 years ago by Martyn Gigg
- Status changed from verify to reopened
- Resolution fixed deleted
comment:11 Changed 9 years ago by Martyn Gigg
It appears this still doesn't quite work on the Mac and with newer versions of glibc (on Ubuntu Oneric Ocelot)
comment:12 Changed 9 years ago by Martyn Gigg
Refs #4204. Add symbol visibility attributes to bpl registry.
Ensures that the boost pythpn type registry is kept hidden within each library on Linux and Mac.
Changeset: 4fa2fb27d6b18a5596681396ed8b2c0714de4b37
comment:13 Changed 9 years ago by Martyn Gigg
Refs #4204. Ensure the same hidden registry for the new API.
Changeset: 6da6fc5edce3c406496e064dc211ffbbcb92b7f0
comment:14 Changed 9 years ago by Martyn Gigg
Refs #4204. Change the Mac to load symbols locally for Python.
Changeset: 4f70264656ffadcc0410de291f337bb9061be59c
comment:15 Changed 9 years ago by Martyn Gigg
Refs #4204. Skip attempting to dynamically load dlopen library also.
Changeset: f649102c81e579113a2a5d88698a190d67e18581
comment:17 Changed 9 years ago by Martyn Gigg
- Status changed from accepted to verify
- Resolution set to fixed
comment:18 Changed 9 years ago by Martyn Gigg
Refs #4204. Fix for the installed version of MantidFramework on Mac
Changeset: cf65373c44b3cebb5810bbede221eaaeab52a71c
comment:19 Changed 9 years ago by Martyn Gigg
Refs #4204. Be slighlty more picky about which file is chosen
Changeset: b8b600532ae223ade80ac030c90311ecc7fc921e
comment:20 Changed 9 years ago by Martyn Gigg
Refs #4204. Fixes MantidPlot startup on Mac
Changeset: 950535c2a7161caa0df175dd66b15e42da4a852c
comment:21 Changed 9 years ago by Martyn Gigg
Refs #4204. Try again to remove the compile flag.
Changeset: 7ec9ba2048344c78b41533ab8e386969ee1e896f
comment:22 Changed 9 years ago by Martyn Gigg
Refs #4204. Fix for the installed version of MantidFramework on Mac
Changeset: cf65373c44b3cebb5810bbede221eaaeab52a71c
comment:23 Changed 9 years ago by Martyn Gigg
Refs #4204. Be slighlty more picky about which file is chosen
Changeset: b8b600532ae223ade80ac030c90311ecc7fc921e
comment:24 Changed 9 years ago by Martyn Gigg
Refs #4204. Fixes MantidPlot startup on Mac
Changeset: 950535c2a7161caa0df175dd66b15e42da4a852c
comment:25 Changed 9 years ago by Martyn Gigg
Refs #4204. Try again to remove the compile flag.
Changeset: 7ec9ba2048344c78b41533ab8e386969ee1e896f
comment:26 Changed 9 years ago by Nick Draper
- Status changed from verify to verifying
- Tester set to Nick Draper
comment:27 Changed 9 years ago by Nick Draper
tested on windows 7 64 bit works beutifully
comment:29 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 5051
So it turns out the static boost libraries on linux are not compiled with fPIC so the library refuses to link to the dynamic module. In any case the fact that we have to use the RTLD flag when loading the python module on linux (to ensure the singletons wire up correctly) means that we can't just statically link anyway.
The solution requires statically linking to boost python (for the old API) AND ensuring that only the Mantid libraries are opened with the RTLD_GLOBAL flag.