Initial note: This type of question is probably better off on either the devel@lists.fedoraproject.org or the packaging@lists.fedoraproject.org mailing lists. The ticketing system is more for making changes to the guidelines.

I took a look at the srpm and it seems like:
* The .py files are fine to install as you are doing.
* The .pyc and .pyo files should not be installed. This shouldn't cause an issue as rpm-build has a script to byte-compile the .py files into .pyc and .pyo.
* The .so files need to be compiled from source
* The mMass/mspy/calculations.so file should be rm'd in %prep
* There's .pyd files so probably, remove the calculations.c file as well and BuildRequire: Cython. Cython will build the .pyd into a .c and then gcc will build that into the .so.
* If I'm reading the setup.py correctly:
* the setup.py build and setup.py install aren't actually doing anything.
* You can remove those lines from the spec file
* You'll have to add your own commands to the spec file to run Cython on the .pyd and then run gcc on the .c.
* Alternately, you can try to modify setup.py to do that building for you.
* neither mspy nor calculations.so are meant to be installed as general purpose python modules by upstream. Therefore you should probably not install calculations.so into %{python_sitearch}. Just installing it into $RPM_BUILD_ROOT%{_datadir}/%{name}/mspy/ should be fine.