Skip to content

Mpi on candide#661

Open
martinkilbinger wants to merge 284 commits intoCosmoStat:developfrom
martinkilbinger:mpi
Open

Mpi on candide#661
martinkilbinger wants to merge 284 commits intoCosmoStat:developfrom
martinkilbinger:mpi

Conversation

@martinkilbinger
Copy link
Copy Markdown
Contributor

@martinkilbinger martinkilbinger commented Oct 31, 2023 •
edited
Loading

Summary

A number of fixes are proposed to get MPI running again (on candide):

  • Version upgrades (gcc -> 9.3.0, openmpi -> 5.0.0, numpy -> 1.22)
  • MPI job script update
  • Some small fixes (module_config_sec was not communicated to clients)
  • Module name fixes in example config file
  • Minor other fixes in modules

Need to wait for pysap and subsequent mccd update, see CEA-COSMIC/pysap#187

Reviewer Checklist

Reviewers should tick the following boxes before approving and merging the PR.

  • The PR targets the develop branch
  • The PR is assigned to the developer
  • The PR has appropriate labels
  • The PR is included in appropriate projects and/or milestones
  • The PR includes a clear description of the proposed changes
  • If the PR addresses an open issue the description includes "closes #"
  • The code and documentation style match the current standards
  • Documentation has been added/updated consistently with the code
  • All CI tests are passing
  • API docs have been built and checked at least once (if relevant)
  • All changed files have been checked and comments provided to the developer
  • All of the reviewer's comments have been satisfactorily addressed by the developer

@martinkilbinger martinkilbinger added the bug Something isn't working label Oct 31, 2023
@martinkilbinger martinkilbinger added this to the P3 re-run 2023 milestone Oct 31, 2023
@martinkilbinger martinkilbinger self-assigned this Oct 31, 2023
@martinkilbinger martinkilbinger mentioned this pull request Nov 8, 2023
Open
@martinkilbinger
Testing dockerfile with only conda env installed
07cafb4
@martinkilbinger
Fixing numpy and astropy versions
92a999c
@martinkilbinger
installation on canfar working: removed most versions from yml file; …
2b6600d 
…added init_canfar.sh script
@martinkilbinger
Dockerfile for image 0:7, 0:8
b6e5c97
@martinkilbinger
removed pysap check for example runner
d3aeccb
@martinkilbinger
Installation of sextractor and psfex with conda
bc66565
@martinkilbinger
script to call curl with canfar contained updated, adding NCORE as ar…
0b3aac6 
…gument
@martinkilbinger
added init scripts for canfar; job_sp added n_smp for further jobs
69317cb
@martinkilbinger
script to call curl for canfar container: added resources
d35bd69
@martinkilbinger
testing input numbers
d93a60e
@martinkilbinger
removed debug msg
fb83a27
@martinkilbinger
numpy error fixed with version upgrade
e33972b
@martinkilbinger
Merge branch 'science_portal_run' of github.com:martinkilbinger/shape…
edbec76 
…pipe-1 into science_portal_run
@martinkilbinger
Getting ready for exclusive-one-tile processing
9aa16be
@martinkilbinger
Added command line option to specify exclusive ID for processing
e5d0335
@martinkilbinger
Merge remote-tracking branch 'origin/exclusive' into science_portal_run
49d8880
@martinkilbinger
curl script exclusive ID
d02347e
@martinkilbinger
Dockerfile conda -> source activate
12e6b63
@martinkilbinger
Merge remote-tracking branch 'origin/docker_image' into docker_image
7d20699 
Conflicts:
	Dockerfile
	environment.yml
	scripts/sh/init_canfar.sh
@martinkilbinger
Merge branch 'docker_image' of github.com:martinkilbinger/shapepipe-1…
1d3df1a 
… into docker_image
@martinkilbinger
comment added
b05f2a5
@martinkilbinger
Updated Dockerfile
45942d5
@martinkilbinger
running with exclusive ID
7d1891e
@martinkilbinger
doc
a7203a3
@martinkilbinger
cleaning up and adding doc strings to vignet maker
7f842a3
@martinkilbinger
added clarifying comments to vignet maker
ee4ef4f
@martinkilbinger
cleaned up curl script
1c8b4cd
@martinkilbinger
canfar scripts adopted to new env; added debug
3210b88
@martinkilbinger
shapepipe_run script adopted to new canfar env
ae97110
@martinkilbinger
setools matplotlib Agg to avoid display error
ebb40b1
@martinkilbinger
ngmix: catching missing keys in postage stamp sqlite database files
ca026ef
@martinkilbinger
combin runs script: added tile_detection
126be96
@martinkilbinger
Merge branch 'v1.4' of github.com:martinkilbinger/shapepipe-1 into v1.4
f60a5f0
@martinkilbinger
cleaned up convery pix script
08d7a7c
@martinkilbinger
summary added special, n_sh as variable
8b4619c
@martinkilbinger
debugging curl canfar
e5c8046
@martinkilbinger
clarified doc on merging star cat
c8f657a
@martinkilbinger
Added checks and improved messages for keys in FITS files
e0eb0f4
@martinkilbinger
debugging curl canfar
ba3a0d8
@martinkilbinger
Pi psfex HDU number changed
ca7f3e5
@martinkilbinger
cosmetic changes
e219400
@martinkilbinger
doc canfar: options sp_local=0,1
34edaaf
@martinkilbinger
split image config more flexible
96ec184
@martinkilbinger
init_curl script: allowing for individual -j 2, 8 (sp_local=1)
45384c9
@martinkilbinger
Merge remote-tracking branch 'upstream/develop' into 651-conda-activa…
925ae5b 
…te-fix-MK
@martinkilbinger
Merge branch '651-conda-activate-fix-MK' into v1.4
c6be9cc
@martinkilbinger
added vos
f1ed5a8
@martinkilbinger
Merge remote-tracking branch 'origin/v1.4' into tiles_202402
f79e0bf
@martinkilbinger
Merge remote-tracking branch 'origin/tiles_202402' into v1.4
e2ec9fa
@martinkilbinger
Merge remote-tracking branch 'origin/v1.4' into cosmos
161d3ca
@martinkilbinger
Merge remote-tracking branch 'origin/v1.4' into mpi
27dc9d2
@martinkilbinger
file_handler updated
805fe03
@cailmdaley
Copy link
Copy Markdown
Contributor

cailmdaley commented Apr 20, 2026

@martinkilbinger — thinking we can close this one? The MPI-on-candide work looks superseded by:

  • Canfar Python Libraries #702 (merged 2026-03-31) — the new CANFAR pipeline with its own async job submission
  • The Docker image as the primary deployment path
  • The branch has also diverged substantially (pre-src/ layout, still uses setup.py, needed pysap PR from 2023)

Happy to close on your nod. If any specific fixes here are still useful, we could cherry-pick them into a fresh small PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfarrens sfarrens Awaiting requested review from sfarrens

Assignees

@martinkilbinger martinkilbinger

Labels

bug Something isn't working

Projects

ShapePipe Dev
Status: To do

Milestone

ShapePipe v1.1.0 Release

Development

Successfully merging this pull request may close these issues.

3 participants

@martinkilbinger @cailmdaley @sfarrens