Path management for different versions of projects

Imagine that you develop a large Matlab program, which uses hundrets of functions. To keep the different modules independent from eachother, the corresponding functions are store in separate folders, e.g.:

 project1\gui\
 project1\calculations\
 project1\export\

During the development your create different version of e.g. the GUI and the export functions and get this hierarchy:

 project1\gui1.01\
 project1\gui1.02\
 project1\calculations\
 project1\export1.01\
 project1\export1.02\
 project1\export1.03\

Which tools are you using (do you recommend) to manage these folders in your MATLAB path? Do your tools consider dependencies, e.g. export1.03\ needs gui1.02\ or higher?

Due to the different folders GENPATH does not match. If you're working on project2\, project3\, etc., using the PATHTOOL and storing the different path strings does not work also, because the versions of the different projects cannot be set independently.

I've written a tool for this job, but it is far too specific and complex to be published. In addition it includes folders of the compiled MEX files for the different Matlab versions and patchs of the toolbox functions (e.g. a TYPECAST and BSXFUN for Matlab 6.5). Using a standard source control system helps to manage the files, but I did not find a comfortable method to care about the set of folders in the MATLAB path. Some of my projects are used for clinical studies and the users must be able to reproduce the results for 10 years after a publication. "Sorry, I cannot get my old source control system to run under Window 14" is not an option.

I think this is a common problem and therefore I'm interested in your solutions.

[Jan]

I'm goind to accept the answer: "Such a function will be included in Matlab 2012a, TMW". :-)

Answer 1

[Bjorn Gustavsson]

In my MyStartup.m I have a call to a function LOADPROJECTS that lets me choose projects, then the paths required gets added to the matlab path. To my LOADPROJECTS I also have EDITPROJECTS, ADDPROJECTS, etc. So I can set up the paths for Project6p5 then unset and set the paths for Project7p2p.

Maybe you're asking for something more extensive, but this covers my current needs to keep separate versions apart. I'd be very interested in hearing what others do.

[Jan]

@Bjorn: Thanks. You answered my question exactly: I want to know how *you* solve the path management for your setup.

[Alvery]

We have set up SVN to automatically generate a .m file to instantiate folder paths when you checkout a workspace. The path includes the "primary" folder + a series of dependent folders. The folder dependencies are stored in a SVN property, and the script recursively looks for the SVN property in all svn:externals folders. svn:externals is a general purpose mechanism within the SVN tool for linking libraries and the like, and links to a specific revision of the library if you want to.

The result is that you can check out the MATLAB code + Simulink models to any location on your computer, and have the path correctly configured.

Answer 2

[Titus Edelhofer]

Hi Jan,

interesting question. And there will be no definite answer ;-). I can only describe my approach:

• generally I'm no fan of having different folders with the same files. I sometimes do it as well (with folders project_xyz_2011_04_01), but at some time it will just fail.
• In my opinion using a good version management system is the only way out. When I have a new version of a project finished, I check out the project, zip it and have it stored separately.
• I admit, that one problem I usually don't have: the integration problem you mention (GUI 1.02 with Compute 2.37). Most of the projects have a coherent versioning.
• This way the folder structure does not change often, and I have no big problems with path setting ...

I'm also curious what others say.

Titus

[Jan]

@Titus: Thanks for this answer. The demands for a version control grow with the size of the projects.

Answer 3

[Daniel Shub]

Your folder naming does not seem to fully capture the power of modern version control systems. If you think of your project as having a version and not the individual components, then you will never have to have folders like gui1.01.

My projects tend to rely on a set of common export and calculations folders. These I version control separately. When I start a new project I import the most recent version of my calculations "project." In this way each project has its own stable copy of the calculations project.

[Jan]

@Daniel: Thanks for this answer. Which modern source control software are you using? Does it consider dependencies between different subfolders? Are you absolutely sure, that this tool can recreate the current state in 10 years?

Using numbered folders has two important advantages: 1. The version appears in error messages automatically. Therefore a bugreport from a customer reveals without any further tricks which version was used. 2. If I ship a new version to a customer it can be installed in parallel to the existing running version without having duplicates for not changed toolbox functions.

Although I'm convinced, that source control system is essential for the development of a large project (I'm talking of >300.000 lines of code + comments and documentation). But for a customer a dull set of numbered folders has the charm of having a minimal complexity.

[Daniel Shub]

I currently use mercurial (HG), although I sometimes wish I used GIT.

I don't know what you mean by dependencies between subfolders. Unlike CVS (and I think SVN), a commit in HG reflects the state of all the files, folders and subfolders. You cannot just rollback a single file. So I think by definition it handles dependencies.

10 years, probably. CVS has a history over 20 years and I believe is still viable. SVN is now over 10 years and is still viable. HG doesn't have the same history, but I am guessing it is popular enough that it should survive. With HG, whenever I release something that I want to archive, I strip the version control info and just backup the files. This means that in 10 years if I have a computer that can run my current OS and the currently installed software (e.g., Windows XP SP2 and MATLAB 2010a), I can reproduce my data (assuming I can get my data to that computer, working zip drives are hard to find nowadays). I worry a little bit about the stuff that I do on a cluster, since I am not sure I will be able to find a cluster with the appropriate OS to handle my compiled code...

If you want numbered folders to make bug reports easier, you can use them in version control systems. Everytime you release a version you can name procject1\gui to project1\gui_vx.y, but I would suggest something like project1_vx.y\gui_vx.y. The key is to prevent replication of the files in the directory structure.

On point 2, installed parallel I think is important. Not duplicating unchanged functions I think is less important. I have seen some projects (e.g., psychtoolbox) which wrap MATLAB around calls to the SVN repository. Assuming a network connection, you never have to ship an update. The update pulls the latest repository. The user then says what version they want to run.

[Jan]

The SVN wrapper sounds very interesting. I'll take a deeper look into the psychotoolbox.

Automatic updates are not wanted by some users of my software: a full test needs 4 to 6 weeks. Therefore the code is freezed and it is preferred to find workarounds for known bugs.

Answer 4

[Arthur]

I manage a similar issue with a product I maintain that uses a lot of common in-house tools that are also used by some unrelated analysis functions. The product and analysis code are kept in separate git repositories. I have packaging & release functions checked in with the product that I use to create reproducible builds for delivery; they wrap a call to git that determines the version number from the latest tag (git describe), build documentation with the version number embedded, run MCC, zip and copy files to a folder for burning to a CD. The product itself looks for git when not isdeployed, and self-reports the current commit ID when running from MATLAB. (It also compares the data files we ship to precomputed MD5 hashes, because we have to worry about "power users" overwriting data files and producing untraceable results.)

I am planning to migrate the shared functionality to separate repositories and use git subtree to bring them in to each repository that needs the functionality. This way, analysis functions will not depend on the product directory. Once that is complete, I will have duplication between the repositories, but there will be no run-time dependencies between them. Changes to the common functions will be propagated between the repositories only on request, so there will not be any surprise updates.

Answer 5

[Jason Ross]

Using a version control system is only part of the answer. You need to also have a release process to produce the finished software package and a means of setting your environment appropriately to run only the files of that release.

One approach could be as simple as copying the files for the "released" project1 and then having a file that sets this environment correctly. Then, project1 is effectively "frozen" in time and you can return to it at any point in the future (*). Of course, in a modern version control system, you can also add in branches, tags, etc to know when each was released and also use it as a means to re-create and patch older versions as needed.

You should also take into account other dependencies you have -- for example, compilers, web servers, drivers and other software bits you might need to re-create the environment at release time. Depending on the complexity and number of interactions in the various software modules, the communatorics of testing changes for past versions may make trying to keep up what version works with what other version to be too complex an undertaking -- for example, if you make a change to package A that's at version 1.0.2 and that is supposed to be compatible with three old releases, you need to test all of those to make sure they still work, give desired outputs, fail in the same way and so on. This can be daunting depending on the size and complexity of your code base.

And, of course, you also need to actually test your releases in a "sterile" environment to ensure that you don't have some dependency that you didn't capture.

• - assuming that the OS you are running can still run the executables. You may need to preserve an appropriate OS image by some means (for example, making an image of the OS and keeping it with your release -- but this opens another can of worms). At some point, the OS vendor will drop support for an old OS, or newer versions of an old OS will no longer keep backwards compatability to an old one. Microsoft has been fairly decent about keeping compatibility, via one means or another -- but the list of caveats gets larger with each older release as new technologies, OS architecture and chip families emerges. I've found that virtualization offers a nice and convenient way to capture OS images and also largely sidesteps the potential hardware/driver issues that exist, since the OS only has to use the virtualization system's drivers, and you don't run the risk of too-modern equipment not having supported drivers that work with a very old OS.

Answer 6

[Jan]

Some FEX submissions:

• FEX: Personal PATHDEF
• FEX: Project manager

Answer 7

[K E]

There is a nice presentation on smart ways to manage directories for a scientific project here. It lays out a consistent approach to naming directories and deciding what should go in them. [Not what your question asked, but it still may be of interest].