How do I use live script (*.mlx) files in source control?

I would like to use the live script files with source control, what is the workflow for that?

Answer 1

[MathWorks Support Team]

How to source control your files within MATLAB:

Since R2014b, you can integrate MATLAB with the Git source control system through the Current Folder browser (see the workflow shown in this video ).

To avoid corruption when doing so, you must register your MATLAB and Simulink file extensions such as .mlx, .mat, .fig, .mlapp, .mdl, .slx, .mdlp, .slxp, .sldd, and .p as binary formats. See more details here:

https://www.mathworks.com/help/matlab/matlab_prog/set-up-git-source-control.html

How to diff/merge your files:

Support for comparing live code (.mlx) files has been shipping with MATLAB since R2018a. A two-way merge tool was also added in R2018b. External version control tools can be configured to use MATLAB Comparison tool using the setup instructions outlined here:

https://www.mathworks.com/help/matlab/matlab_prog/customize-external-source-control-to-use-matlab-for-comparison-and-merge.html

Future improvements:

We're committed to making the Live Editor features work more seamlessly with source control – and we're also committed to providing an enhanced experience with plain text MATLAB files. We're working hard on both of these fronts, so stay tuned for future updates!

[James]

Do you have any instructions for configuring source control to use the MATLAB differencing tool? What is the correct command line?

[Simon Silge]

As Steven Lord has correctly written below you can now customize your source control system to use the MATLAB diff and merge tool as explained on this page. Also, source control is now integrated into the MATLAB surface, the doc page is here. There is also a quick introduction video.

[Vasco]

From what I understand until now is that there are two kind of contents in a mlx file:

1. Text input (code, text blocks). These are all pure text. This is very important to keep under version control.
2. Output blocks, which can contain anything including figures and movies. This can be a lot of data, and is typically not very important to have under version control.

Why not have include 1 as plain text in the mlx file, and add 2 as zipped and base64 encoded (or a utf8/utf16 equivalent like that) on separated lines? This way code reviewing can be done with standard tools (like bitbucket). The main thing that can happen is that a code cell gets two outputs (instead of one). The live editor could detect this and show a warning.

With this file format is is also relativity easy to create a mlx file without output.

[Walter Roberson]

Why not have include 1 as plain text in the mlx file, and add 2 as zipped and base64 encoded

The use case for mlx files is that the user should only have to copy one file and that would bring all relevant parts with it. This is the same strategy as for Simulink .slx files.

.mlx files are zip files that are mostly XML files, using the same kind of directory structure used by Microsoft .xlsx files -- OpenXML structures.

Any inserted code or text or controls are encoded as xml.

Any inserted equations are primarily encoded as xml, but a mathml equivalent of them is also included.

Any inserted images have coded by relationship identifiers, and the appropriate .rels file (which is more XML but with a different extension) gives a directory reference to a binary copy of the image -- the image itself is included in the .mlx structure, not just a filename reference to the image.

You could potentially compare by unzip the .mlx file and comparing files. You might, however, have some difficulty with relationship identifiers being different -- I see no attempt in the files to allocate a GUID to relationships so as to be able to compare them even if you did something like insert a new control in the middle.

[Olaf Bousche]

Although I like live notebooks I have the same gripe about sorce control as most people. I have found a partial solution. Just delete all output and commit the clean copy. That works but sometimes matlab still adds internal stuff that does not show up in the difftools. Based on this I want to propose the following improvements:

• Don't save stuff just because you can. Adding stuff llike "saved with version" or something of that nature is a killer for version control. If you want to have something like that let it go away with "delete all output"
• If there are differences that are relevant, have the difftools be aware of changes other than just code so they can be inspected and accepted if required.

Hope this helps a bit

[Walter Roberson]

@Olaf Bousche

The save version is important in order to be able to understand the file format, which evolves. I have never yet found a non-trivial file format that did not change after it was first introduced.

[Steven Lord]

@Olaf Bousche Rather than clearing all the output before checking the file in, have you tried configuring your SCM to use the tools we provide for comparing and merging live scripts (under "How to diff/merge your files" in the MathWorks Support Team's answer)? If so and if you found that solution unsatisfactory could you say a little more about why?

[Markus Leuthold]

@Steven Lord

• you cannot see any diffs on github
• you cannot git blame
• you cannot make one commit per topic by staging single lines which belong to a certain topic

All of these are important aspects of git which I don't want to loose

Answer 2

[Teemu Sillanpää]

So with the change from readable content (text) to binary, live scripts / to achieve .m files with documentation built in feature is totally useless in normal working environment (>1 user).

Answer 3

[Markus Leuthold]

This is going into a completely wrong direction. Source code files must be text files. The toolchain does not only consist of Matlab, but many other such aus Github etc... which become unusable with these mlx files.

Unfortunately, there are already new features which are only available in the Live editor, but not in the classic editor or command window, such as tab completion via functionSignatures.json

[Walter Roberson]

I used to rail against web pages with images because "Source code files must be text files!", and you couldn't copy and paste the representation of images. I lost that fight, Tools evolved.

I also appear to have lost the fight against CSS...

[Michael Abboud]

Hi Markus, I'm a developer on the MATLAB Editor team and I'm glad to share that this Auto Completion functionality from the Live Editor is now available in plain text (.m) MATLAB files in the R2021b pre-release of the MATLAB Editor!

Many of the other coding tools from the Live Editor (Block selection, Refactoring, etc.) are also now available in plain text (.m) MATLAB files – along with a refreshed look and feel – starting with the R2021b pre-release which launched last week.

We think this is a big step forward, and we're excited for you to give it a try! If you're able to install a new version of MATLAB, I'd encourage you to try out the R2021b pre-release and send us feedback:

https://www.mathworks.com/downloads/prerelease

Answer 4

[Richard Crozier]

No you cannot use live scripts with any of the major version control systems such as svn, git or mercurial. They are a binary format and not compatible with these text difference based systems. Versioning the files will be possible in a very basic sense but none of the standard diffing and merging commands will work effectively, and none of the standard GUI tools for comparing versions will work as the files will simply appear as binary blobs.

You may be able to set up your source control system on one or two machines to work as specified in the Matlab docs, but no remote services such as Github or Bitbucket will support this.

Answer 5

[Teemu Sillanpää]

Has this "readable content being source controlled as binary" been fixed already?

The whole idea of readable code is the ability to commit/diff changes in readable form in source controlled environment. Now zipping the .XML to a .xml without the source control understanding the nature of the content is purely wrong approach. Live scripts can be written/converted also in .m but this is not intuitive process in the GUI environment - ending up having two source controlled files (risk of being unsync) between .m and .mlx. Third option would be that Mathworks provides GIT tools which understand the .mlx as readable content. Or the .mlx files being converted to ASCII (.m ish) before source controlled.

Any of these properties implemented yet in the R2017b or coming R2018a?

[Walter Roberson]

"Any of these properties implemented yet in the R2017b or coming R2018a?"

No. It is unlikely that the representation as a zip'd series of xml files will be changed any time soon.

[Steven Lord]

As of release R2016b you can customize your source control system to use the diff and merge tools included in MATLAB for binary .mlx files. See this page in the documentation for instructions on how to configure your system.

Answer 6

[Jan Kappen]

Any updates on that? Workarounds? Plugins for Github or GitLab? Mathworks is effectively destroying any kind of collaboration, --automatic testing-- (wrong, see below) and code reviewing.

I'd stick to .m files if the editor wasn't that bad :(

Even juypter notebook files are in text format... The Notebook file format — nbformat 5.1 documentation

[Steven Lord]

Plugins for Github or GitLab?

Perhaps GitHub Actions will be of interest to you?

Mathworks is effectively destroying any kind of collaboration, automatic testing and code reviewing.

How so?

Using the third general source control topic on this documentation page you can use the MATLAB Comparison Tool on Live Scripts, MAT-files, or Simulink models as long as your source control tool allows you to specify tools to use to diff and merge files with certain extensions.

You can write tests using the testing frameworks and execute them in CI systems.

You can also use the MATLAB Comparison Tool inside MATLAB for code reviews.

[Jan Kappen]

Please ignore "automatic testing" in my rather provocant statement, that is not true, apologizes. CI & automated tests works well. Github Actions are the right direction, unfortunately, my company uses Bitbucket and Gitlab - no way to change that. Runners for CI work on local machines so better than nothing.

The part about collaboration and code review however remains true. Most of my developer colleagues (partially including myself) prefer not to use the Matlab GUI for code reviews (or anything else). This kind of work is done in the web interface where the pull request are shown or in good text editors that properly support git, pull requests, all kind of different languages, such as VSCode. There is no way to comment .mlx files line by line, or see incremental changes per commit online. One always has to checkout the whole commit and call the Matlab comparison tools (which have non-obvious interfaces and limited functionality - compare it with GitLens in VSCode and you know what I'm talking about ;)

Colleagues without having Matlab installed (or when no flex license is free) have no chance to do code reviews. That's certainly not the way to go.

What about supporting VSCode plugins like Add feature to support .mlx file(a matlab live script file) · Issue #115 · Gimly/vscode-matlab (github.com)? (I'm not the maintainer)

Thanks!

[Walter Roberson]

@Jan Kappen

Your company should stop using .mlx files if you have those needs. If that forces you to move to a different programming language, then that forces you to move to a different programming language.

.mlx files will never be plain-text files. If, hypothetically, Mathworks were to create similar functionality in plain-text files, then they will not use the .mlx file extension for such a hypothetical file.

I repeat: As long as .mlx files exist, they will always be a zipped structure of XML... exactly like .xlsx files are and ECMA-376 (ISO/IEC 29500) and Open Document Format (ODF) are. No amount of plugins or UI improvements will ever change that, so if your workflow requires that you use plain-text files, then .mlx files will always be incompatible with that.

[Jan Kappen]

Dear @Walter Roberson,

I'm afraid we indeed will stop using .mlx files though the idea is good. It's a too big source of error and slows down the development process, just as easy as that.

I'd like to continue using .m files, but the editor (auto completion, look & feel) is worse than the live editor. That means, I'd have to code in the live editor and save it as .m file to convert it to .mlx next time I (or my other 50 colleagues working with Matlab) want to make bigger changes where refactoring and auto completion features are needed. That's not a working workflow. Maybe commit hooks or one of these packages to convert .mlx to markdown can help: livescript2markdown: MATLAB's live scripts to markdown - File Exchange - MATLAB Central (mathworks.com) Live Script To Markdown Converter - File Exchange - MATLAB Central (mathworks.com)

Btw, as several people already mentioned the same issue in this thread, my needs are not that uncommon - in fact the majority of review workflows in software development I'm aware of are using web interfaces. And I wonder which company has not the same or at least similar requirements as ours has. These requirements automatically occur when a code review workflow combined with GIT (does not "like" binary data) is established. Which is standard.

Jupyter Notebooks work very similar as Livescripts, they contain code, text, pictures, latex and meta data and can be a) textually compared, b) rendered in GitHub, GitLab and Bitbucket, so it seems to be possible. I understand that .ipynb files are json, but as you state, .mlx is basically xml (zipped), so there should be ways to at least render them on the fly.

A comparison with .xlsx is not appropiate here imo. Microsoft has its own style of versioning and reviewing files via O365 and OneDrive/SharePoint. These files are simply not meant to be pushed to an VCS. On the other hand, Matlab .mlx files contain code. Code that often lived in .m files before. This should be reviewable outside Matlab.

I'm not asking to change the file format at all, I'm asking for helper to get around the fact that they cannot be rendered outside of Matlab.

Thanks.

[Walter Roberson]

xlsx files can certainly contain code. Macros are code. People write big complex spreadsheets in xlsx. No substantive difference. If an organization is willing to put a xlsx file under version control, then it should be willing to put an mlx file under version control.

[Jan Kappen]

Walter, while the .xlsx paragraph was obviously not the main point (the main point is: binary files don't belong to standard VCS like git, it's that simple, and Mathworks should think about that) I don't think any company puts .xlsx files under git version control, at least if they have a choice. .m files were good for version control, .mlx are not.

Any idea if features from .mlx editor (I don't care about the embedded plots, I mean refactoring and auto completion) will be usable for .m files in the future?

[Walter Roberson]

And I am saying that MATLB used a standard binary container format -- standard as in International Standard -- that is used for spreadsheets, word processors, image handling, scientific data. They did not "roll their own": they used a common standard. You would think that version control systems would have caught up to that container format by now.

[Michael Abboud]

Hi all, I'm a developer on the MATLAB Editor team and wanted to share a few thoughts on this topic.

First off, I'm really excited to share that many of the coding tools from the Live Editor (Auto Completion, Refactoring, etc.) are now available in plain text (.m) MATLAB files – along with a refreshed look and feel – starting with the R2021b pre-release which launched last week. We think this is a big step forward, and we're excited for you to give it a try! If you're able to install a new version of MATLAB, I'd encourage you to try out the R2021b pre-release and send us feedback:

https://www.mathworks.com/downloads/prerelease

On the topic of plain text (.m) vs Live Editor (.mlx) files, we know that different tools work well for different folks. We're committed to making the Live Editor features work more seamlessly with source control – and we're also committed to providing an enhanced experience with plain text MATLAB files. We're working hard on both of these fronts, so stay tuned for future updates!

[Jan Kappen]

Dear @Michael Abboud,

I've receied the prerelease notification yesterday and was more than happy to read that the live editor features were ported back to .m files, this is awesome, thank you!

Thank you also for your last paragraph, it's good for us to know that work is being done and that a potential gap in means of source control is identified.

So thanks!