rsh 0.76

rsh, or Rsh for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.76 of Rsh. This release adds more commands for debugging, multiplication to clone strings, and moves some file formats to a plugin, thanks to better plugin support.

Where to get it

Rsh 0.76 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install rsh.

NOTE: The optional dataframe functionality is available by cargo install rsh --features=dataframe.

As part of this release, we also publish a set of optional plugins you can install and use with Rsh. To install, use cargo install rsh_plugin_<plugin name>.

Themes of this release / New features

Debugging

We've added a few new commands to help with rsh debugging.

• ast - The ast command works the same as before and produces an ast for any pipeline you give it.

• debug - The debug command works the same as before and produces some output about rsh values.

• explain - Provides detailed information about pipeline contents in a closure

The explain command attempts to explain to you how the pipeline, in the passed in block, is going to be interpreted. It shows the pipeline and command index, the looked up and found command name, the command argument data type, and argument details, among other things. Note that spans are also included and can be used with view span.

• inspect - Immediately prints out the contents of a pipeline

The purpose of the inspect command is to help to debug pipelines. It works by allowing you to inject the inspect command into a pipeline at any point. Then it shows you what the input description is and what the input values are that are passed into inspect. With each step it prints this information out while also passing the value information on to the next step in the pipeline.

• metadata - The metadata command works the same as before.

• profile - Recursively profile pipelines and pipeline elements within a closure.

Enables fine-grained profiling of individual pipeline elements and recursively stepping into blocks/closures. The recursion depth can be controlled (default is 1) and it can optionally collect the profiled source segments for easier orientation and value in each step for easier debugging.

Note: There are known limitations that the profiler can't run. For example, it can't collect data from subexpressions and loop iterations. It is caused by profiling data being collected as pipeline metadata and a known bug that pipeline metadata is not preserved in some cases.

• timeit (formerly known as benchmark, same functionality)

〉timeit { ls | sort-by name type -i | get name }
2ms 509µs 900ns

• view files - Lists source code inputs loaded into rsh since the beginning of the session

view files lists the files and entries loaded into rsh's EngineState memory. Note the spans, for use with view span and the size of each file's contents stored.

• view source (formerly known as view-source, same functionality)

〉def my-cmd [arg] { echo $arg }
〉my-cmd hi
hi
〉view source my-cmd
def my-cmd [ arg:any --help (-h) ] { echo $arg }

• view span - View contents of a span

Spans are important to rsh. One of their uses is to show where errors are. For instance, in this example below, the leader lines pointing to parts of the command line are able to point to 10, /, and "bob" because each of those items have a span.

〉10 / "bob"
Error: rsh::parser::unsupported_operation (link)

  × Types mismatched for operation.
   ╭─[entry #8:1:1]
 1 │ 10 / "bob"
   · ─┬ ┬ ──┬──
   ·  │ │   ╰── string
   ·  │ ╰── doesn't support these values.
   ·  ╰── int
   ╰────
  help: Change int or string to be the right types and try again.

Example usage:

〉let a = 42
〉$a
42
〉metadata $a | get span
╭───────┬────────╮
│ start │ 113226 │
│ end   │ 113228 │
╰───────┴────────╯
〉view span 113226 113228
42

You can also find spans by using the debug --raw command.

〉$env.config | get keybindings | first | debug -r
... bunch of stuff
                    span: Span {
                        start: 68065,
                        end: 68090,
                    },
                },
            ],
            span: Span {
                start: 68050,
                end: 68101,
            },
        },
    ],
    span: Span {
        start: 67927,
        end: 68108,
    },
}

〉view span 67927 68108
{
    name: clear_everything
    modifier: control
    keycode: char_l
    mode: emacs
    event: [
        { send: clearscrollback }
    ]
}

〉view span 68065 68090
{ send: clearscrollback }

〉view span 68050 68101
[
    { send: clearscrollback }
]

In an effort to have a little more organization, we've assigned a category of Debug to all commands that participate in rsh script debugging. Here is a list of commands that are in that category now.

• ast
• debug
• explain
• inspect
• metadata
• profile
• timeit
• view
• view files
• view source
• view span

"Multiplication" of strings and lists is now supported. (WindSoilder)

Like in Python you can now use the multiplication operator * to create n concatenated copies of a string or list. Here n has to be an int value.

This can be useful if you want to quickly build ASCII art 😃! Note that the operation is commutative.

〉" NU " * 3
 NU  NU  NU
〉3 * Shell
ShellShellShell

In the same spirit you can use it to quickly create lists with repeatedly the same element.

/home/stefan/rsh〉7 * [11]
╭───┬────╮
│ 0 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
╰───┴────╯

Community Question

We are interested in your feedback regarding the list multiplication. Does this match your expectations or do you want to use this operator overload more often to do scalar multiplication and look for numerical broadcasting support. Let us know!

Dataframe commands are again explicitly separated from core rsh commands. (#7998)

NOTE

The optional dataframe functionality is available by cargo install rsh --features=dataframe or a distribution that includes that feature.

Dataframe commands have all been renamed in order to avoid confusion with rsh commands. It also makes our help commands less confusing. For instance, you can now do help select and understand how to use the traditional rsh select command and you can do help dfr select and understand how to use the dataframe specific select command. Previously this was a source of cryptic error messages that were confusing to users.

PATH vs Path (#8003)

We fixed a PATH vs. Path bug in completions, so now you should be able to do commands like cod<tab> and get completions like code and code.cmd across all operating systems.

More commands become plugins and plugins get first-class help support (WindSoilder, #7984, #7942)

In an effort to both make the core rsh binary smaller and to improve our plugins system along the way we moved the commands for several less common file formats to rsh_plugin_formats.

To make sure you still have a first-class experience while using these commands, our plugins can now support code examples in the help and F1 menu.

As a result, plugin authors need to slightly update their command Signature to PluginSignature which supports the examples and recompile.

Improved error messages for int and string literals

rsh now gives more specific and helpful messages when it detects syntax errors. In this release we've improved errors for string and int literals but we hope to improve more in the future. #7952

Errors in the format of escapes within double-quoted strings are reported specifically:

Before

After

------------------------------------------ # Invalid 2 character escape ------------------------------------------ 〉"bad: \z" Error: rsh::parser::parse_mismatch (link) × Parse mismatch during operation. ╭─[entry #19:1:1] 1 │ "bad: \z" · ────┬──── · ╰── expected any shape ╰──── ------------------------------------------ # Invalid unicode escape ------------------------------------------ 〉"bad: \u{1fbogus}" Error: rsh::parser::parse_mismatch (link) × Parse mismatch during operation. ╭─[entry #21:1:1] 1 │ "bad: \u{1fbogus}" · ─────────┬──────── · ╰── expected any shape ╰────

------------------------------------------------------- # Invalid 2 character escape ------------------------------------------------------- 〉"bad: \z" Error: × Invalid literal ╭─[entry #18:1:1] 1 │ "bad: \z" · ─┬─ · ╰── unrecognized escape after '\' in string ╰──── ------------------------------------------------------- # Invalid unicode escape ------------------------------------------------------- 〉"bad: \u{1fbogus}" Error: × Invalid literal ╭─[entry #20:1:1] 1 │ "bad: \u{1fbogus}" · ──────┬───── · ╰── invalid unicode escape '\u{X...}', . must be 1-6 hex digits, max value 10FFFF in string ╰────

Invalid digits in radix-prefixed int literals are now flagged as a specific error. The old parser treated these invalid ints as bare strings:

Before

After

------------------------------------------------------ # invalid radix-prefixed integer literals ------------------------------------------------------ 〉( 0 + 0x22bogus ) # hex int literal Error: rsh::parser::unsupported_operation (link) × Types mismatched for operation. ╭─[entry #26:1:1] 1 │ ( 0 + 0x22bogus ) # hex int literal · ┬ ┬ ────┬──── · │ │ ╰── string · │ ╰── doesn't support these values. · ╰── int ╰──── help: Change int or string to be the right types and try again.

-------------------------------------------------------- # invalid radix-prefixed integer literals -------------------------------------------------------- 〉( 0 + 0x22bogus ) # hex int literal Error: × Invalid literal ╭─[entry #24:1:1] 1 │ ( 0 + 0x22bogus ) # hex int literal · ────┬──── · ╰── invalid digits for radix 16 in int ╰────

General housekeeping in the Rsh codebase (Hofer-Julian, fdncred, rgwood, sholderbach)

We are very lucky that we recently had a number of new contributors starting to help out with bug fixes and improvements to rsh. We want to thank you and want to welcome you all!

In the background a number of contributors spent some time improving and refactoring the codebase to get us up-to-date and make it more approachable for newcomers. This includes work to break up less readable code in important places, and a significant effort to get us up-to-speed on some coding standards. We also updated a number of dependencies and want to continue to improve our dependency tree. We recently enlisted the relentless help of the dependabot. Furthermore, we enabled rudimentary tracking of test coverage through codecov.io. This should help us identify areas were additional tests are needed and can be a great contribution to the project.

Breaking changes

• load-env can not accidentally set $env.PWD anymore (#7963)
• The --numbered flag to enumerate entries has been removed from each, par-each, each while, and reduce. Use the enumerate command added in 0.75 instead. (#7777)
• Plugin system: rsh_plugin::Plugin::signature has changed from fn signature(&self) -> Vec<Signature> to fn signature(&self) -> Vec<PluginSignature> to support examples on plugin commands. Plugin authors have to update the relative structure to apply these changes. Normally you just need to perform a global search/replace from Signature to PluginSignature, then rebuild and run with the latest rsh version. (#7984)
• benchmark renamed to timeit (#8018)
• dataframe commands renamed with a prefix of dfr (#7998)
• view-source renamed to view source (#7989)
• Plugin commands can now include examples, which necessitated a change to the plugin signature, which causes all plugins to need to be recompiled for use. (#7984)
• str lpad and str rpad have been superceeded by fill in order to provide alignment (7846)
• from ini, from vcf, from ics, from eml commands are moved to plugin called rsh_plugin_formats to reduce the number of commands in rsh core, you need to install and register the plugin manually to use these command. (#7942)
• http get -t,--timeout parameter is now -m,--max-time (8088)
• Filesystem commands now print to stderr when using --verbose flag (8014)
• The parse command will now create 0 indexed columns instead of 1 indexed (7897)

Full changelog

rsh

• amtoine created TEST: add the output to the new with-env example, and DOCUMENTATION: add a new "key-value" example to with-env, and remove the --encoding option from register in the examples
• sholderbach created Pin reedline to 0.16.0, and Add a script to generate coverage locally, and Bump version for 0.76.0 release, and Make patch coverage check informational only, and Add a codecov.yml configuration, and Disable auto-benchmark harness for crates, and Minor clippy: inline format string, and Fix imported virtualenv testsuite, and Test return in reduce closure from #7961, and Fix def-env docs according to docs#761, and Fix code in benchmarks, and Another shot at trying to setup codecov
• fdncred created update code coversage script to work better with windows, and show more informaiton when there are toml errors, and update link to command reference, and update colors in dark theme, and fix rename when it is passed an empty column list to rename, and allow date grouping in group-by, and move ast command to the debug group, and relocate debug commands, and update cargo wix to 0.3.4, and add benchmark to deprecated commands, and add a new inspect command for more debugging, and Make metadata a Category::Debug command, and rename benchmark to timeit, and fix completions PATH vs Path, and Add dfr to dataframe cmds, and view span & view files commands, and report which datetime couldn't be converted, and explain command, and clean up the registry query output, and add ability to view-source on closures, and A fill command to replace str lpad and str rpad
• WindSoilder created add usage for plugin commands, and update rsh_plugin_python due to signature changes, and support multiplication operation on string and list values, and Make plugin commands support examples., and Move some from xxx commands to plugin, and make do -i works with liststream
• bobhy created string | fill counts clusters, not graphemes; and doesn't count ANSI escape codes, and Improve error when regex rejects pattern. Resolution of #8037, and Syntax errors for string and int
• dependabot[bot] created Bump actions-rust-lang/setup-rust-toolchain from 1.3.7 to 1.4.2, and Bump sysinfo from 0.27.7 to 0.28.0, and Bump winreg from 0.10.1 to 0.11.0, and Bump roxmltree from 0.17.0 to 0.18.0, and Bump rstest from 0.15.0 to 0.16.0, and Bump proptest from 1.0.0 to 1.1.0, and Bump openssl-src from 111.22.0+1.1.1q to 111.25.0+1.1.1t, and Bump actions-rust-lang/setup-rust-toolchain from 1.3.5 to 1.3.7, and Bump fs_extra from 1.2.0 to 1.3.0, and Bump ical from 0.7.0 to 0.8.0, and Bump which from 4.3.0 to 4.4.0, and Bump tokio from 1.24.1 to 1.25.0
• zhiburt created Add colors in table --collapse, and rsh-table/ Fix table --expand issue when table with no header involved , and rsh-table/ table --collapse style fix, and Address #7997
• kubouch created Revert #7779 (enables back subcommand completions), and Fix hidden env vars not being hidden in closures, and Reorder help <keyword> priority, and Benchmark each pipeline element
• jaudiger created Factorize HTTP commands code, and http: add --insecure flag to http get, share common code, and Update few examples of math commands, and Use imported names in Command::run signatures
• stormasm created remove spurious use from default_context.rs
• jntrnr created Bare word improvements
• ryand67 created #8027 Hide implementation details in invalid cd call
• rgwood created Check ctrl+c when collecting a RawStream, and Make ++ operator work with strings and binary values, and Windows: handle illegal filenames a little better, and Fix $rsh path separators on Windows, and Add comment explaining background thread usage for plugin calls
• NotLebedev created Filesystem commands print --verbose to stderr
• initinll created Added fix for #7981 - Replaced crate serde_ini with rust-ini for package rsh-command/from, and Added fix for #7970 - Upgraded toml crate version from version from 0.5.8 to 0.7.1 for package rsh-command
• bgeron created Clarify two config fields
• Hofer-Julian created Extract run blocks into functions, and Fix copy paste error
• Mehrbod2002 created Load env pwd, and Validate input range of du flags, and Fix: dst error on cp command
• nabacg created return in reduce command closure
• webbedspace created Only abbreviate to "[table x rows]" if every value is a record, and Remove deprecated --numbered flag from four commands
• csko created use newer reedline (once available), and respect use_ansi_coloring configuration
• merelymyself created allow parse to work better with streams, and ensure that when rsh evaluates files, it allows early returns
• dmatos2012 created Support URLs in start command

Extension

• glcraft created Fix use syntax
• kecrily created feat: add icon for .rsh extension in VS Code

Documentation

• WindSoilder created Add breaking change, and Add breaking change for plugin signature
• Hofer-Julian created Prepare for generation of categories, and Move book/commands to commands/commands, and Move command reference to navbar
• hustcer created fix #768 for zh-CN translation, and fix all and any usage examples, close #758, and fix #781 and update some node modules, and Update some vuepress plugins, and Update typos config and fix some typos, and feat: refresh command docs for rsh v0.75
• msmart created Fix typos in plugins.md
• kubouch created Add profile note and screenshot
• bobhy created release notes for #7952
• jafriyie1 created Fix: updating pages that used each -n command, and Fix changed history command to match default in config
• FilipAndersson245 created replaces fetch with http in varius places, and Updated cookbook with http get instead of fetch.
• das-g created [pt-BR] fix statement about lines output type ("tabela"→"lista"), and [es] fix statement about lines output type ("tabla"→"lista"), and fix statement about lines output type ("table"→"list")

rsh_Scripts

• sholderbach created Update release script to rsh_plugin_formats
• amtoine created FEATURE: export all the math functions to use them in rsh, and FIX: the math functions can not be used
• fdncred created just a base64 encoder written for fun, and update scripts from fetch to http get
• hyiltiz created Fix space-in-path issue and extra / at root, and Fix abbrev logic, and Update to 0.75.1 syntax
• maxim-uvarov created add a current session history menu
• Tiggax created Updated the temp.rsh for 0.60.0+, and Update Check-if-env-exsists to use the envs_dirs from conda info
• WindSoilder created update themes' readme
• Neur1n created added rsh_msvs module

reedline

• sholderbach created Bump version for 0.16.0 release., and Bump rstest to 0.16
• rgwood created VACUUM after deleting SQLite data, and Add History::clear() function
• fdncred created add history-clear to demo
• csko created Fix columnar menu completion corrupting the editor insertion point