After watching the successes and frustrations of Hernan, John, and others as they worked to release ROMS 3.0, it is obvious that the collaboration technology, or our application of it, is probably not yet optimized. Over the next year, we should think about what method(s) would be easiest and most productive for improving the code without endangering progress. I don't think we have to worry much about losing code...the bigger issue seems to be losing time when people have to undo or redo things.
I think the greatest advance that SVN has brought us is VISIBILITY...being able to see and assess changes others have made in the code is a huge advantage, and svn, Trac, WinMerge, and other tools facilitate that. I also think the system has improved our ability to protect the code from accidental mistakes or lost fixes. Although the system isnt foolproof, it is better than we had before.
I think we should encourage potential developers to have branches in the repositories and used them as they modify the code for their own purposes. If nothing else, it will make it easier for them to assess how changes by others might affect their own code, and it would certainly make it easier to incorporate their changes into the mainstream model, if they wanted to do that
We also need to work on sharing the workload and ownership. If potential contributors do not get an opportunity to provide input into the design of ROMS or feel like they can put effort in and be rewarded by seeing that code appear in releases that they can use and others can applaud, they won't put the effort in. We have to somehow harness the work of people who download, modify, and are never heard from again. We have to think about how to do this.
Among other things, we should preferentially use the Discussion forum to debate these things...this started out as an e-mail, but I realized it should have a broader audience.
Here are a few questions...some small and short term, others more philosophical and provocative:
1) Does the CSTM have a tagged release in the ROMS_SED repository that is equal to Hernan's 3.0 (version 37)? Should we?
2) How do we keep up with small fixes (like Hernan's changes from tagged release to version 38). When do we make an official tagged release with accumulated small fixes? What is the role, really, of a tagged release? Maybe the tagged release should be the most up-to-date stable version...the one we recommend for download. There might be several of these in the days and weeks after a big new release (like 3.0...would generate 3.01, 3.02, etc.).
3) Does Rutgers have Trac running for Hernan's version? That make it easier to see what changes have been made, and can integrate code browsing, wiki documention, and svn with issue tracking.
4) Should the official ROMS Documentation be under Trac Wiki? Pros and cons?
5) For now, should we encourage users (of all interests) to download the "official" Rutgers version, but encourage all sediment-related developers to work from the ROMS_SED repository? That would mean that anybody interested in modifying the sediment code would download from the ROMS_SED trunk and could have their own (private, if desired) branch in the ROMS_SED repository, regardless of whether they are partners in the NOPP CSTM project. (I strongly advocate this, but know others do not....feedback?)
5) Is the any advantage to having Rutgers ROMS, UCLA ROMS, ROMS_SED, ROMS_Agrif, and ROMS_whoever be hosted in the same repository?
I'm thinking that by having a common ROMS site, people could see the differences, share progress, not repeat things, build more flexible and useful tools, and maybe coalesce the codes. In the spirit of the C in CSTM, that project could support a joint website to help keep the differences in the various versions visible, if not reconciled.
Collaboration, releases, SVN, Trac, flavors of ROMS...
Collaboration, releases, SVN, Trac, flavors of ROMS...
Chris Sherwood, USGS
1 508 457 2269
1 508 457 2269
Good topics, Chris!
I know Rutgers already has at least three svn repositories with differing permissions, for trunk, branches, and CSTM. Hernan first told me about the trunk, then his branch, so I tried to "svn switch" from one to the other. Imagine my surprise when svn told me I couldn't do that.
Now that we have a central repository for ROMS itself, I'd like to see a push to get some of the related tools out there too. People ask here on the forums about the old Fortran forcing program, for instance. My colleagues and I haven't used it in years. Instead we have Matlab scripts we all use and hate, hoping to replace them someday, someday with something Open Source. I'm sure the other ROMS branches have plenty to contribute along those lines too, not to mention other models - one NOAA gang is using Delft3d for grid generation (not that it's free in any sense).
I know Rutgers already has at least three svn repositories with differing permissions, for trunk, branches, and CSTM. Hernan first told me about the trunk, then his branch, so I tried to "svn switch" from one to the other. Imagine my surprise when svn told me I couldn't do that.
Now that we have a central repository for ROMS itself, I'd like to see a push to get some of the related tools out there too. People ask here on the forums about the old Fortran forcing program, for instance. My colleagues and I haven't used it in years. Instead we have Matlab scripts we all use and hate, hoping to replace them someday, someday with something Open Source. I'm sure the other ROMS branches have plenty to contribute along those lines too, not to mention other models - one NOAA gang is using Delft3d for grid generation (not that it's free in any sense).
-
jo.irisson
- Posts: 5
- Joined: Wed Mar 28, 2007 1:23 pm
- Location: Rosenstiel School of Marine and Atmospheric Sciences
Hello,
I am very new to ROMS and, at the moment, I only read the forums to gather experience from the success and failure of others. This is my first message but I thought I would provide insights from my involvement in what I think is a very healthy Open Source Software project (Inkscape for those who know it http://www.inkscape.org)
First of all having the code in SVN offers great potential for collaboration but also implies a non-trivial amount of work to keep it clean and usable when the repository layout complexifies (tags, branches etc.) and this is usually not something that can be managed by a community so it requires a few people with good knownledge of SVN to administrate it in addition to their usual workload IMHO. I personnaly keep all my personal code in SVN and find it very useful , especially to remember what I did on different projects, but all branches I do are dead-born because it requires too much work to keep them synchronized with the rest to be worth it.
In ROMS, I don't think that there is such a marked difference between experienced users/developers and basic users, so maybe the whole tag-release cycle (which is quite energy and time consuming) is pointless: pointing users to a specific revision may be enough. Tagging a major release from time to time would only provide greater visibility of the advances in the code, which is good for advertising and self-satisfaction but does not add any particular functionality compared to "svn checkout http://blablablabl@135".
One particular way to use branches I find very effective is when only part of the code is to be modified in a specific branch: a developer can switch only a subset of the files to the branch (with svn switch) and keep the rest in the trunk. Let say there are three directories A, B and C. C is switched to branches/ROMS_SED while A and B stay in the trunk. "svn update" on this mixed working copy will get changes from the trunk to A and B and changes from the branch to C. This garantees that changes in C are compatible with the latest version of the trunk and reduces the trouble of merging back the changes from the branch to the trunk to the files in C (and this can be done on a file by file basis also).
If the different codes are similar enough currently to recreate the branches from one particular version which is currently under SVN (i.e. diff ROMS vs. ROMS_SED, use ROMS as the trunk, copy ROMS/trunk to ROMS/branches/ROMS_SED and apply the diff there), and if code exchanges between all versions is expected to happen regularly then managing those as branches is worth it. If the codes are already so different that they cannot share history on at least some of the files, making them branches will probably be difficult and will complicate things.
I hope this helps.
I am very new to ROMS and, at the moment, I only read the forums to gather experience from the success and failure of others. This is my first message but I thought I would provide insights from my involvement in what I think is a very healthy Open Source Software project (Inkscape for those who know it http://www.inkscape.org)
First of all having the code in SVN offers great potential for collaboration but also implies a non-trivial amount of work to keep it clean and usable when the repository layout complexifies (tags, branches etc.) and this is usually not something that can be managed by a community so it requires a few people with good knownledge of SVN to administrate it in addition to their usual workload IMHO. I personnaly keep all my personal code in SVN and find it very useful , especially to remember what I did on different projects, but all branches I do are dead-born because it requires too much work to keep them synchronized with the rest to be worth it.
In Inkscape great effort is put in keeping the trunk usable at all times for real-life work. This allows experienced users to test it and brings quicker feedback to the developers. Every 6 months or so the trunk developement is frozen for a month, only bug fixes are added and in the end a major release is tagged and distributed to the bulk of Inkscape's users who just want to draw and not to bother with development. As soon as a major release is out, development continues on the trunk. Point releases are independant from the trunk and only add bug fixes to the tagged version.csherwood wrote:2) How do we keep up with small fixes (like Hernan's changes from tagged release to version 38). When do we make an official tagged release with accumulated small fixes? What is the role, really, of a tagged release? Maybe the tagged release should be the most up-to-date stable version...the one we recommend for download. There might be several of these in the days and weeks after a big new release (like 3.0...would generate 3.01, 3.02, etc.).
In ROMS, I don't think that there is such a marked difference between experienced users/developers and basic users, so maybe the whole tag-release cycle (which is quite energy and time consuming) is pointless: pointing users to a specific revision may be enough. Tagging a major release from time to time would only provide greater visibility of the advances in the code, which is good for advertising and self-satisfaction but does not add any particular functionality compared to "svn checkout http://blablablabl@135".
Having many different versions of the same code definitely calls for branches indeed but merging changes back and forth between the trunk and branches should be done very often otherwise it probably will never happen. Branches in SVN are conceptually nice but prove to be difficult to manage. From what I know, this seems to be less painful with distributed revision control systems (like Bazaar, SVK etc.) in which each developer checkout is a branch itself but I do not have personal experience with them.csherwood wrote:5) For now, should we encourage users (of all interests) to download the "official" Rutgers version, but encourage all sediment-related developers to work from the ROMS_SED repository? That would mean that anybody interested in modifying the sediment code would download from the ROMS_SED trunk and could have their own (private, if desired) branch in the ROMS_SED repository, regardless of whether they are partners in the NOPP CSTM project. (I strongly advocate this, but know others do not....feedback?)
5) Is the any advantage to having Rutgers ROMS, UCLA ROMS, ROMS_SED, ROMS_Agrif, and ROMS_whoever be hosted in the same repository?
I'm thinking that by having a common ROMS site, people could see the differences, share progress, not repeat things, build more flexible and useful tools, and maybe coalesce the codes. In the spirit of the C in CSTM, that project could support a joint website to help keep the differences in the various versions visible, if not reconciled.
One particular way to use branches I find very effective is when only part of the code is to be modified in a specific branch: a developer can switch only a subset of the files to the branch (with svn switch) and keep the rest in the trunk. Let say there are three directories A, B and C. C is switched to branches/ROMS_SED while A and B stay in the trunk. "svn update" on this mixed working copy will get changes from the trunk to A and B and changes from the branch to C. This garantees that changes in C are compatible with the latest version of the trunk and reduces the trouble of merging back the changes from the branch to the trunk to the files in C (and this can be done on a file by file basis also).
If the different codes are similar enough currently to recreate the branches from one particular version which is currently under SVN (i.e. diff ROMS vs. ROMS_SED, use ROMS as the trunk, copy ROMS/trunk to ROMS/branches/ROMS_SED and apply the diff there), and if code exchanges between all versions is expected to happen regularly then managing those as branches is worth it. If the codes are already so different that they cannot share history on at least some of the files, making them branches will probably be difficult and will complicate things.
I hope this helps.
I definitely think having Trac available for the Roms_rutgers SVN repository would be a good idea. There is already a Trac system at myroms.org for the ROMS CSTM repository, and it's proven very useful. At the very least, it's a great way to browse the source, examine recent code changes, and compare SVN versions right from your browser. It's also a bug tracking system, which I also think would be very useful.
For a graphic example, the bug report posted under "ROMS bugs"
viewtopic.php?t=508
about a missing continuation character in rhs3d.F was not replied to by a developer. So how we know if it has been fixed?
A quick check of the CSTM Trac source browswer tells me that the last change was 18 hours ago by Arango:
Then if I click on the "view diffs" link for rhs3d.F, I can see that the bug has indeed been fixed:
Pretty nice!
Rich
For a graphic example, the bug report posted under "ROMS bugs"
viewtopic.php?t=508
about a missing continuation character in rhs3d.F was not replied to by a developer. So how we know if it has been fixed?
A quick check of the CSTM Trac source browswer tells me that the last change was 18 hours ago by Arango:
Then if I click on the "view diffs" link for rhs3d.F, I can see that the bug has indeed been fixed:
Pretty nice!
Rich
Okay, I see from Hernan's post in the ROMS bugs discussion that the Trac site for ROMS does exist:
https://www.myroms.org/projects/src
Thanks Hernan & David!
If you haven't already checked out the capabilities on "browse source", it really is nice.
Should we submit bugs now as "new ticket" on the Trac site as opposed to the ROMS Bugs discussion list?
Thanks,
Rich
https://www.myroms.org/projects/src
Thanks Hernan & David!
If you haven't already checked out the capabilities on "browse source", it really is nice.
Should we submit bugs now as "new ticket" on the Trac site as opposed to the ROMS Bugs discussion list?
Thanks,
Rich
-
arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
Yes, we want everybody to post bugs and algorithm suggestions in the Trac website. This is a good place to keep all bugs in a same place in terms of tickets. It also gives the user the opportunity to check which tickets have been resolved.Should we submit bugs now as "new ticket" on the Trac site as opposed to the ROMS Bugs discussion list?
The ROMS Trac website has been running since last February. Yes, the Trac web pages contain a lot of nice information.
We are in the process of switching to the new ROMS version and I have some questions concerning administration: Imagine that on a cluster of our university, there are three people using ROMS and making setups, all working on the same project, but all going into a slightly different direction (one is setting up analytical setups without wind stress, one includes analytical wind forcing and uses a slightly different domain, the third one forces with non-analytical data, and so on). Everybody would like to know what the others do, and include their code if appropriate (with permission of the others).
In addition, everybody would like to be able to incorporate the bug fixes of the roms/toms group in all of their setups.
I read a few lines about subversion, and the idea came up that every user creates a branch for a new run (in parameter-sensitivity studies there can be a lot of runs). But to create branches, we need a repository. So what I'm thinking is:
1)checking out Roms from the roms/toms-group server to get a working copy on the cluster (let's call it ROMS_vanilla)
2)create a local repository from ROMS_vanilla on our cluster.
3)let every user create branches in the local repository for every specific run she sets up.
4)run svn update in ROMS_vanilla once in a while to get bugfixes made by the roms/toms-group
5)commit these changes to the trunk of our local repository once in a while to give the three users a chance to incorporate the bugfixes in their branches.
Sounds a little too complicated. Is there a better way to do this? Any suggestions?
Of course we don't intend to use our local repository in a way that causes problems relating to 'Method C' in
http://www.myroms.org/index.php?page=License_ROMS.
Our repository would only be accessable for the scientists working on the project.
Thanks in advance
--stef
By the way, how can you create a repository from a working copy?
In addition, everybody would like to be able to incorporate the bug fixes of the roms/toms group in all of their setups.
I read a few lines about subversion, and the idea came up that every user creates a branch for a new run (in parameter-sensitivity studies there can be a lot of runs). But to create branches, we need a repository. So what I'm thinking is:
1)checking out Roms from the roms/toms-group server to get a working copy on the cluster (let's call it ROMS_vanilla)
2)create a local repository from ROMS_vanilla on our cluster.
3)let every user create branches in the local repository for every specific run she sets up.
4)run svn update in ROMS_vanilla once in a while to get bugfixes made by the roms/toms-group
5)commit these changes to the trunk of our local repository once in a while to give the three users a chance to incorporate the bugfixes in their branches.
Sounds a little too complicated. Is there a better way to do this? Any suggestions?
Of course we don't intend to use our local repository in a way that causes problems relating to 'Method C' in
http://www.myroms.org/index.php?page=License_ROMS.
Our repository would only be accessable for the scientists working on the project.
Thanks in advance
--stef
By the way, how can you create a repository from a working copy?
-
jo.irisson
- Posts: 5
- Joined: Wed Mar 28, 2007 1:23 pm
- Location: Rosenstiel School of Marine and Atmospheric Sciences
I am afraid that what you want to do is not possible, at least not this way. Indeed you cannot create a local repository (with history and all) from a working copy and still get the updates from the global one. In fact if you create a new repository from your working copy, your working copy will just be considered as any other folder, its svn URL will be the one of your local server and svn update on this will not bring you the changes made to the official ROMS repository.
There are solutions however:
- sync the official repository to your local machine (not the working copy, the entire repository) but this requires you to have access to the server where the official repository is hosted and use rsync or something. this is probably difficult
- use SVN in a more convoluted (but powerful!) way:
1. create an _empty_ repository on your server (with tags, branches and trunk, all empty). see the SVN book on how to create a repository
2. in trunk, make an svn external call. svn external allows you to have code from another SVN repository inside your own project. In your case, your whole trunk will be an external project: the official ROMS trunk. Then svn update will bring you the modifications made to ROMS trunk
3. create all the branches you want in this local repository and merge changes from the trunk from time to time
NB: I would advise you to use mixed working copies as described in my earlier post here or in the switch section of the SVN book. basically you keep most of your working copy in the trunk and only switch the files that you need to modify to the branch. this allows you to get updates from the trunk regularly without the need to merge. I think this is particularly appropriate in your case.
Cheers,
There are solutions however:
- sync the official repository to your local machine (not the working copy, the entire repository) but this requires you to have access to the server where the official repository is hosted and use rsync or something. this is probably difficult
- use SVN in a more convoluted (but powerful!) way:
1. create an _empty_ repository on your server (with tags, branches and trunk, all empty). see the SVN book on how to create a repository
2. in trunk, make an svn external call. svn external allows you to have code from another SVN repository inside your own project. In your case, your whole trunk will be an external project: the official ROMS trunk. Then svn update will bring you the modifications made to ROMS trunk
3. create all the branches you want in this local repository and merge changes from the trunk from time to time
NB: I would advise you to use mixed working copies as described in my earlier post here or in the switch section of the SVN book. basically you keep most of your working copy in the trunk and only switch the files that you need to modify to the branch. this allows you to get updates from the trunk regularly without the need to merge. I think this is particularly appropriate in your case.
Cheers,
You dont really need your own repository
Hi Stef,
The new version of ROMS has a directory structure that reduces the need for changing code with each application. Unless you are really developing new components of the model, you can work by updating the from the main repository...in fact, this would keep you in closer touch with any bug fixes or improvements.
I recommend the following strategy: users svn copy the repository trunk to a convenient local spot. After that, they svn update regularly. For code related to a particular application (cppdefs, analytical, etc), they store that code in an application directory, and use the build.sh or build.bash scripts to include that code. For real developments, make code/algorithm improvements in your copy. Updates will not overwrite that code, but may warn you of a conflict, which is good...that means you need to check to see if changes in the ROMS trunk affect your development. When your code improvements are complete and run in an otherwise up-to-date version of ROMS, please submit a TRAC ticket and include the new code improvements...then they will be migrated to the repository where we can all enjoy them.
The new version of ROMS has a directory structure that reduces the need for changing code with each application. Unless you are really developing new components of the model, you can work by updating the from the main repository...in fact, this would keep you in closer touch with any bug fixes or improvements.
I recommend the following strategy: users svn copy the repository trunk to a convenient local spot. After that, they svn update regularly. For code related to a particular application (cppdefs, analytical, etc), they store that code in an application directory, and use the build.sh or build.bash scripts to include that code. For real developments, make code/algorithm improvements in your copy. Updates will not overwrite that code, but may warn you of a conflict, which is good...that means you need to check to see if changes in the ROMS trunk affect your development. When your code improvements are complete and run in an otherwise up-to-date version of ROMS, please submit a TRAC ticket and include the new code improvements...then they will be migrated to the repository where we can all enjoy them.
Chris Sherwood, USGS
1 508 457 2269
1 508 457 2269
Thank you for the replies!
Ok, we will try to isolate the user-modified parts of the code and avoid using svn for anything else than checking out a working copy from the roms/toms-group repository for every user.
This seems to be easy and gives every user a chance to learn how to use svn.
When we get to know svn better and see how it can help to organize the confusion on our local network arising from user-to-user communication, we can still switch to a more sophisticated approach, like the one outlined by jo.irisson.
--stefan
Ok, we will try to isolate the user-modified parts of the code and avoid using svn for anything else than checking out a working copy from the roms/toms-group repository for every user.
This seems to be easy and gives every user a chance to learn how to use svn.
When we get to know svn better and see how it can help to organize the confusion on our local network arising from user-to-user communication, we can still switch to a more sophisticated approach, like the one outlined by jo.irisson.
--stefan
Relating to this discussion I want to point to a different post. The following is *very* instructive:
viewtopic.php?t=622
Thanks!
viewtopic.php?t=622
Thanks!