A new promising technology: Spy

.

I read what is available on the web about the new language SPy. It is clearly a very interesting and attractive project!

Some times ago, I wrote a serious note on Mojo and I feel that I will have to write a similar note on SPy.

However, I currently have many questions on the project and Antonio Cuni, the creator of SPy, plans to write “a series of posts in which he will try to give a deep explanation of SPy”. Therefore, I’m going to wait for these notes and then write my own post to explain what I understand on SPy from my own point of view of Python user focussed on scientific computing and Python teacher .

Here, I write a short post on how I understand the SPy project with few questions. My goal is not to ask Antonio to answer to these questions and remarks any time soon. It is more to express how I currently receive the project and to present the points that I don’t understand yet.

SPy lang: what is it?

First, few words about my current understanding of what is SPy. SPy is a very new Python-like language (a Python variant):

  • One of the way to described SPy is “a better Cython”:

    • it is much more similar to Python than to C

    • it has better tooling

    • it has a better type system

    • it will be possible to create Python extensions (like with Cython) but also programs independent of any interpreter

  • “Static Python” but dynamic at compile time

  • Interpreted or compiled (through transpilation to C)

  • Implemented in Python

  • Relatively simple implementation

  • Serious implementation and good tooling (compiler/debugger)

  • Focused on “in-browser” execution (without interpreter), through WASM/Emscripten.

  • Garbage collector (no Rust-like borrow checker)

    • much simpler and easier than Rust -> less safety.

    • “unsafe” a bit better that C (context and Zig-like defer ?).

  • Should be great and useful in terms of start-up time!

Few questions

Roadmap?

Unless I missed it, the project does not have any explicit roadmap.

What are the planned next steps? What is needed to get a first “distributed and versioned version” (so that SPy can be used without its repository)? And a first “stable version”?

What is planned for the SPy standard library? Mimic Python for things like os , pathlib , argparse , re , … ?

Faster interpreter

Now, few questions about the speed of the interpreter. Currently, the SPy interpreter is implemented in Python and runs with CPython. It is therefore very very slow (like much slower than CPython!).

Are there some plans to accelerate the SPy interpreter?

A “simple” solution to accelerate interpreted SPy would be to use PyPy instead of CPython. However, PyPy is not in a good shape in terms of maintenance and also cannot be used in the browser.

Another “evident” response would be to use SPy for the SPy interpreter. However, when we think a bit about the technical challenge for that, it seems … challenging; and the language needs to be much more mature.

SPy is a language and a command-line program (the SPy interpreter/compiler). Rewriting all dependencies of the CLI tool in SPy (Click, Typer, Ninja, …) is not easy so I guess SPy won’t be fully written in SPy in a short time.

However, once SPy can be distributed and can produce Python extensions, it could be possible that the Python program spy uses Python extensions written in SPy (or actually just independent internal programs, so that we don’t even need extensions). In particular, the SPy interpreter could be written in SPy. Is it the plan?

Another similar question is about the relationship and similarity with RPython, which can be described as a compiler framework to write interpreters (like PyPy) and give them a “meta-JIT” for free (no need to write the JIT). SPy takes a lot of inspiration from RPython. Will it also be possible to write a meta-JIT with SPy or this aspect of RPython is not an objective for SPy?

Sharing code between Python and SPy?

SPy uses files with extensions .spy. Even though SPy code looks a lot like Python, it is not currently possible to share code between Python and SPy. Currently, this is not convenient for benchmarks, for which we need to have and maintain 2 versions of the code (Python and SPy) even for the parts that are compatible. It seems to me that it can also be in the future an issue for project developers of SPy+Python libraries.

Will it be possible to share code usable from Spy and from Python?

SPy libraries… Usable from Python?

Will it be possible to transform Python libraries into SPy libraries also usable from Python?

We can for example consider Typer, Click or Ninja. It seems to me that theoretical, many Python libraries like those once could also be SPy libraries since the dynamical aspects can be put at “compile time”. How could it work?

It is also interesting to consider that SPy will need an implementation of the Python array API standard for arrays in CPU usable from SPy. It would be nice if this implementation could be also usable from Python.

Note that if such thing exists one day, most Python functions using Numpy could be compiled with SPy, meaning that SPy would also become “a better Pythran”.

Packaging and distribution of Spy libraries

Do you think it could be possible to use PyPI to distribute SPy (the interpreter/compiler) and SPy libraries? What about related modern tools like UV?

Alternatively, SPy could focus on conda-forge or another specific conda channel (like Mojo)?

Note that Mojo uses mainly Pixi , which could also be very good and adapted for SPy.

However, since SPy is so much related to Python, it could make sense to use both PyPI and conda (like Python).

Should SPy projects use a pyproject.toml file or something else? Mojo don’t do that but it seems that it could be reasonable for SPy.

What about testing

Pytest and Coverage are great for Python. Can they be used for SPy? What is planned regarding testing of SPy projects?

Parallel computing?

CPython recently got a new free-threading mode and subinterpreters in the standard library. What about SPy? What is planned regarding parallel computing?

SIMD, GPU and Mojo

Mojo is another new language inspired by Python and other modern languages. Even though Mojo shares some characteristics with SPy (Python-like, dynamic/static and interpreted/compiled), the two languages are quite different. They clearly have different primary purposes. Mojo is focused on AI applications. Moreover, the Modular/Mojo/Max project is technically much more ambitious (first language based on the compilation framework MLIR) and its implementation is therefore much more complex and expensive. In contract, SPy compiler is based on the C stack and one can guess that its development should cost an order of magnitude less.

That said, SPy can learn from Mojo development by adopting some particularities/APIs or in contrast take different choices.

Mojo is in particular strong for its ability to express algorithms in a general way and use SIMD instructions. What about SPy?

More generally, how SPy can be compared to Mojo? In which aspects SPy is better than Mojo?

What about SPy and GPU computing?

Type system

Even though huge progresses have been done, Python’s type system is known to be much less good than type system of modern static languages, like in particular Rust.

What about SPy type system? How can it be described and compared to Python and Rust type systems?