rsh 0.73

rsh, or Rsh for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.73 of Rsh. This release includes new math commands, an interactive data viewer, and many command refinements.

Where to get it

Rsh 0.73 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install rsh.

NOTE: The optional dataframe functionality is available by cargo install rsh --features=dataframe.

As part of this release, we also publish a set of optional plugins you can install and use with Rsh. To install, use cargo install rsh_plugin_<plugin name>.

Themes of this release / New features

PLEASE NOTE: Boolean && and || have changed

The boolean && is now and and the boolean || is now or. These were deprecated in the 0.72 release and have now been removed. Existing scripts will need to be updated to the new syntax to run in 0.73.

Experimental interactive explore command (zhiburt)

This release includes a new experimental command called explore for viewing Rsh data in an interactive UI. Some things to try:

• pipe a large table to explore (ex: ls | explore) and use explore as a fancy pager
• run explore, then type :try and press the Enter key to enter a mode where you can run commands inside explore

explore is highly experimental and we expect it to change in the future. Please report any issues you discover.

New math commands (sholderbach)

With this release we include a larger number of math commands for real valued math such as trigonometric functions and logarithms. The goal is to remove the math eval command that operates on strings instead of proper rsh expressions.

Constants

• math pi
• math tau
• math e

Trigonometry and hyperbolic functions

• math sin
• math cos
• math tan
• math sinh
• math cosh
• math tanh
• math arcsin
• math arccos
• math arctan
• math arcsinh
• math arccosh
• math arctanh

Logarithms

• math log
• math ln

〉math pi | math cos
-1
〉math e | math ln
1
〉[16 8 4 2] | math log 2
[4 3 2 1]

Changes to commands with predicates (kubouch)

any, all, skip until, skip while, take until, and take while now accept a closure instead of a row condition. For example

[[status]; [UP] [UP]] | all status == UP

becomes

[[status]; [UP] [UP]] | all {|el| $el.status == UP }

This makes them slightly more verbose but it is a part of an effort to refactor our parser to allow defining concrete grammar rules for the Rsh language. Row condition is currently accepted only in the where command which becomes a parser built-in command (like use or let).

New filter command and simpler where (kubouch)

We found the -b flag of where quite confusing and decided to remove it in favor of a new filter command. Both filter and where have similar functionality but where now only accepts a row condition and filter accepts a closure. Before:

[{a: 1} {a: 2}] | where -b {|x| $x.a > 1}

After:

[{a: 1} {a: 2}] | filter {|x| $x.a > 1}

Why is it useful to have two commands doing the same thing? Because with the filter command, you can store the closure in a variable:

let cond = {|x| $x.a > 1}
[{a: 1} {a: 2}] | filter $cond

On the other hand, where is more concise ([{a: 1} {a: 2}] | where a > 1) but you can't store its condition in a variable. The choice is yours!

New command uniq-by (raccmonteiro)

To complement uniq which can identify unique or duplicated values in a collection or report the number of occurrences for a particular entry, we now have uniq-by. It supports filtering a table by entries appearing in a particular column.

〉 [[fruit day]; [apple monday] [apple friday] [Apple friday] [apple monday] [pear monday] [orange tuesday]] | uniq-by fruit
╭───┬────────┬─────────╮
│ # │ fruit  │   day   │
├───┼────────┼─────────┤
│ 0 │ apple  │ monday  │
│ 1 │ Apple  │ friday  │
│ 2 │ pear   │ monday  │
│ 3 │ orange │ tuesday │
╰───┴────────┴─────────╯

Mutable data structures can now have their inner values mutated using = (webbedspace)

If a variable has been defined as mutable using the recently-added mut keyword, you can now deeply mutate its inner values using =.

〉 mut a = { b: 1 }
〉 $a.b = 2
〉 $a.c = 3
〉 $a | to nuon
{b: 2, c: 3}

This syntax enhancement was added primarily to make modifying $env.config during a normal session much easier. Now, $env.config is considered inherently mutable. You can change a single config value like so:

$env.config.table.mode = compact_double

...and it will remain in effect for the rest of the session.

More sophisticated coloring of data values using closures (webbedspace)

The color_config config record has new functionality: instead of specifying a single color name for all values of a given type, you may now alternatively provide a closure that dynamically computes a color for each individual value. The closure takes, as input, a single value of the given type, and must produce either a string value representing a color, or a { fg, bg, attr } record (the same as what color_config already accepts as a color value). This feature can be used to provide important visual distinctions between ranges of values in data.

Here are some examples.

filesize: {|e|
	if $e == 0b { 'dark_gray'
	} else if $e < 1mb { 'cyan_bold'
	} else { 'blue_bold' }
}

This causes all filesize values to be colored using dark_gray if they equal 0b, cyan_bold if they are less than 1mb, and blue_bold otherwise. This means that, in ls output, empty files can be more easily distinguished from non-empty ones.

bool: { if $in { 'light_cyan' } else { 'light_gray' } }

This colors true in light_cyan, and false in light_gray. This can be useful when browsing a large table of booleans.

The themes in the default config.rsh file have been updated with further examples of these closures.

In certain situations (most notably, during a single ls call that isn't piped to anything else) rsh will parallelize the execution of these closures. As such, you cannot expect that they will run in the same order as each value appears in the output data.

WARNING

Currently, closures are only supported for output values - they do not work with color settings that begin with shape_, such as shape_literal. They also do not work with the color configuration for the new explore command. Only values inside of tables are highlighted using closures.

RISC V binary release

Starting with 0.73, rsh now provides a RISC V for Linux as part of the set of release binaries.

Breaking changes

• save no longer overwrites files by default, you must use the new -f/--force flag to overwrite an existing file (#7262)
• split row command will retain an empty string if splitted string is empty. (#7413)
• split list now properly removes the separator in all positions (#7355)
• The --show-created-paths flag has been replaced by --verbose from mkdir.
• fetch: the --output, --bin, --append flags have been removed, use fetch with save to save fetched page to a file. (#7468)
• changes to how get works with deep cell paths (#7480)
• mkdir's -s/--show-created-paths has been renamed to -v/--verbose for better consistency (#7462)
• && is now and and || is now or
• any, all, skip until, skip while, take until, take while now take only a closure as an argument.
• where now takes only row condition argument, the -b flag has been removed (use filter instead of where -b).

Full changelog

rsh

• jt created bump to 0.73, and Turn off cd abbreviations by default, and Revert "Add pipeline operators to help", and Revert "Pipeline operators: && and ||", and Add pipeline operators to help, and Pipeline operators: && and ||, and Remove and/or from 'help operators', and Fix input redirect for externals, and Add OneOf shape to fix else, and Improve empty pipelines, and couple minor updates to xml deps
• sholderbach created Remove unused deps or move to devdeps, and Use rsh-path correctly in rsh! test macro to make dev-dependency transitive, and Add example showing first class closure to do, and Add arbitrary base math log, and Add math tau, and Fix math e usage text
• rgwood created Revert "into cellpath command (#7417)", and explore tweaks Round 1, and Fix cell path when getting columns of non-records, and Remove unnecessary echo uses from examples, and Add helper method to check whether ctrl+c was pressed, adopt it, and Handle ctrl-c in uniq and uniq-by, and Fix streaming page missing newline, and Make hook execution stream instead of collecting, and Remove use of deprecated actions-rs/cargo GH action, and Fix watch for block+closure split, and Pin CI jobs to Ubuntu 20.04, and Make SQLite queries cancellable, and Improve error message for illegal filenames on Windows, and Overhaul schema command, remove database name, and Handle mixed LF+CRLF in lines, and Handle ctrl-c in RawStream iterator
• merelymyself created prevent panic with format command, and ensure get doesn't delve too deep in nested lists, and let UnknownFlag error list out available flags, and add interact-once switch to rm, and ensure error in else is forwarded appropriately, and fix external completions; add a caret when there is overlap, and Add quotes to hash file autocomplete, and Make seq return a ListStream where possible
• zhiburt created rsh-explore/ Fix configuration issues, and Patch explore 4, and Patch after fix after fix 7380, and Fix #7486, and Try to fix #7380, and Try to fix #7338, and rsh-explore/ A few things, and rsh-explore/ A few fixes., and Deliver a few fixes for explore command
• webbedspace created Fix encode base64 type signature and examples, and Help messages: edit various instances of "block" to "closure", and Fix du error message, and Tweak "Cannot convert {x} to a string argument" error in run_external, and Reduced LOC by replacing several instances of Value::Int {}, Value::Float{}, Value::Bool {}, and Value::String {} with Value::int(), Value::float(), Value::boolean() and Value::string(), and Edited help text and examples in explore for readability, and Allow $env and mutable records to be mutated by = (closes #7110), and Rename $env.config.explore_config to $env.config.explore (for consistency with $env.config.ls, $env.config.table etc.)
• fdncred created add xterm color names to ansi --list, and add new plugins to script, and add --long flag to history command for sqlite history, and add missing shapes to default_config, and sort enums add missing items to parse_shape_name, and remove example missed from an earlier refactor, and bump to dev build v0.72.2, and add input_output_types() to ansi gradient, and add :q! alias to explore command, and add background colors to the ansi command
• pinjeff created replace lazy_static with once_cell
• andrasio created (register-plugins.rsh): Filter out files ending with .d on systems other than windows., and (rsh_plugin_python): Send back the signature required fields.
• WindSoilder created refactor: introduce is_external_failed to PipelineData, and simplify try logic, and break for, loop, while execution when external command runs to failed, and remove output, append, bin flag from fetch command, and in for, loop, while, auto print final value in each iteration, and make split row works like python and rust ways, and fix semicolon doesn't work for some commands, and fix split list when separator is the first element of list
• hustcer created Add riscv64 binary release target, and add input_output_types() to benchmark,cd and config reset, and Use setup-rust-toolchain for release workflow
• raccmonteiro created some filesystem command signatures, and mkdir change flag -s to -v, and ++= appendAssign operator (#7346)
• Kangaxx-0 created Add config mutation tests, and into cellpath command, and fix upsert index of zero, and Add comments for rsh syntax shape
• kubouch created Replace row conditions with closures in commands, and Simplify FILE_PWD setting in 'overlay use', and Add FILE_PWD environment variable when running 'rsh script.rsh', and Make env-related tests more resilient, and Fix tab not working in vi editor mode, and Move 'where' to parser keywords; Add 'filter' command, and Fix where -b flag
• metacoma created fix docker build
• stormasm created remove redundant code mentioning ToCsv
• nibon7 created kill: don't show signal example on windows
• JohnJohnstone created fix menus in default config
• bgeron created Fix documentation for merge

Extension

• fdncred created add and and or so they highlight properly

Documentation

• WindSoilder created add breaking change for fetch, and add breaking change for split row, and add command signature section
• webbedspace created [release-notes-0.73] Add color closures and deeply mutable data structure features
• rsteube created custom completion: added external_completer
• zhiburt created Add a description file for explore command
• Tengs-Penkwe created Correct command, and Correct command examples
• raccmonteiro created mkdir --show-created-paths flag replaced, and typo fix
• sin created Fix error in book/dataframes.md
• hustcer created feat: optimize make_docs script for better performance, and chore: Upgrade some outdated dependencies
• thara created Fix dead links
• rgwood created Remove old features from install instructions
• Kissaki created Fix typo in 0.71 release notes
• sholderbach created Autoformat book/types_of_data.md

rsh_Scripts

• jt created Since last release script, and Fix TWiN script, and Switch to 'and' and 'or'
• xlittlerag created fixed get-weather celsius loop
• 1Kinoti created fix up command for v0.72.0
• WindSoilder created fix custom completions arg names
• fdncred created change for to each in oh-my.rsh
• ehdevries created Replace deprecated operators in panache-git
• Decodetalkers created feat: add example for starship and shell_space

reedline

• jmoore34 created Make DefaultPrompt configurable