rsh 0.77

rsh, or Rsh for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.77 of Rsh. This release adds reworked aliases, more consistent timestamp handling, reworked XML support, and more.

Where to get it

Rsh 0.77 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install rsh.

NOTE: The optional dataframe functionality is available by cargo install rsh --features=dataframe.

As part of this release, we also publish a set of optional plugins you can install and use with Rsh. To install, use cargo install rsh_plugin_<plugin name>.

Themes of this release / New features

Reworked aliases (Breaking changes!) (kubouch)

Aliases have been a constant source of panics and growing code complexity as a result of trying to patch the panics. In this release, we re-implement aliases from scratch. Instead of replacing spans of expressions, aliases are implemented as another type of command, quite like extern is used to implement known externals. Alias is a command that wraps another command call. As a result, in some cases, aliases do not behave exactly the same as before. Here are the key facts:

• Alias can only alias another command call. For example, alias la = ls -a works, but the following does not:
  • alias foo = "foo"
    • "foo" is not a command call, use alias foo = echo "foo" instead
  • alias lsn = (ls | sort-by type name -i)
    • subexpression is not a command call, use a custom command instead
• Alias cannot alias command named the same as the alias. E.g., alias ls = ls -a is not possible currently, and gives an unhelpful error message. We plan to fix this as soon as possible and in the future we aim for this to work.
• Some parser keywords are not allowed to be aliased. Currently, overlay commands can be aliased but the other parser keywords can not. We can add support for aliasing more parser keywords in the future.

If some of the above is too limiting for you, the old aliases are still unchanged and available as old-alias. Just change alias to old-alias and it should work the same as before. If there are no more problems with the new alias implementation, and we manage to iron out the recursive alias issue, we will remove old-alias in the next release, otherwise, we'll keep it around longer.

More consistent timestamp handling (bobhy)

Simplified conversion between rsh date type and unix timestamps (#8244).

rsh now standardizes on representing a Unix timestamp as a number of nanoseconds relative to the unix epoch 1970-01-01 00:00:00 +0000 (UTC). Since the timestamp is stored in a (64 bit signed) rsh int type, this limits the range of dates that can be represented to approximately 21-sep-1677 through 11-apr-2262.

In prior versions, rsh attempted to extend the range of representable dates by allowing multiple resolutions of timestamps (seconds, milliseconds as well as nanoseconds) to be stored and relied on arbitrary range check heuristics to disambiguate the value intended. However, there were bugs in the checks and incorrect results could be produced.

With this change <int> | into datetime assumes the input is a number of nanoseconds and can never produce a date outside this range:

〉"7fffffffffffffff" | into int -r 16 | into datetime
Fri, 11 Apr 2262 23:47:16 +0000 (in 239 years)
〉("7fffffffffffffff" | into int -r 16) * -1 | into datetime
Tue, 21 Sep 1677 00:12:43 +0000 (345 years ago)

The timestamp epoch is the standard unix epoch. Note the timezone is UTC/GMT:

〉0 | into datetime
Thu, 01 Jan 1970 00:00:00 +0000 (53 years ago)

<datetime> | into int can now produce an error if the input is outside the supported range:

〉1492-10-12 | into int
Error: rsh::shell::incorrect_value

  × Incorrect value.
   ╭─[entry #51:1:1]
 1 │ 1492-10-12 | into int
   ·              ────┬───
   ·                  ╰── DateTime out of timestamp range 1677-09-21T00:12:43 and 2262-04-11T23:47:16
   ╰────

And finally, although not strictly required by the above fix, <date> | date to-record and <date> | date to-table now have a nanosecond field containing the subsecond residue of the input value (however it was produced).

〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-record
╭────────────┬───────────╮
│ year       │ 2262      │
│ month      │ 4         │
│ day        │ 11        │
│ hour       │ 23        │
│ minute     │ 47        │
│ second     │ 16        │
│ nanosecond │ 854775807 │
│ timezone   │ +00:00    │
╰────────────┴───────────╯
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-table
╭───┬──────┬───────┬─────┬──────┬────────┬────────┬────────────┬──────────╮
│ # │ year │ month │ day │ hour │ minute │ second │ nanosecond │ timezone │
├───┼──────┼───────┼─────┼──────┼────────┼────────┼────────────┼──────────┤
│ 0 │ 2262 │     4 │  11 │   23 │     47 │     16 │  854775807 │ +00:00   │
╰───┴──────┴───────┴─────┴──────┴────────┴────────┴────────────┴──────────╯

New XML format (NotLebedev)

New format for xml data created and accepted by from xml and to xml commands (#7947).

Commands from xml and to xml now use format where each xml entry is represented by a single {tag: <tag name> attributes: <tag attributes> content: [<child entries>]} record. Special xml entries also use this record, replacing irrelevant fields with null for easier use.

Reading some simple xml:

〉'<release>
    <project repo="https://github.com/radhesh1/rsh">rsh</project>
    <version>0.77</version>
    <message>Now with better xml!</message>
  </release>' | from xml
╭────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ tag        │ release                                                                                                     │
│ attributes │ {record 0 fields}                                                                                           │
│            │ ╭───┬─────────┬───────────────────────────────────────────────┬───────────────────────────────────────────╮ │
│ content    │ │ # │   tag   │                  attributes                   │                  content                  │ │
│            │ ├───┼─────────┼───────────────────────────────────────────────┼───────────────────────────────────────────┤ │
│            │ │ 0 │ project │ ╭──────┬────────────────────────────────────╮ │ ╭───┬─────┬────────────┬─────────╮        │ │
│            │ │   │         │ │ repo │ https://github.com/radhesh1/rsh │ │ │ # │ tag │ attributes │ content │        │ │
│            │ │   │         │ ╰──────┴────────────────────────────────────╯ │ ├───┼─────┼────────────┼─────────┤        │ │
│            │ │   │         │                                               │ │ 0 │     │            │ rsh │        │ │
│            │ │   │         │                                               │ ╰───┴─────┴────────────┴─────────╯        │ │
│            │ │ 1 │ version │ {record 0 fields}                             │ ╭───┬─────┬────────────┬─────────╮        │ │
│            │ │   │         │                                               │ │ # │ tag │ attributes │ content │        │ │
│            │ │   │         │                                               │ ├───┼─────┼────────────┼─────────┤        │ │
│            │ │   │         │                                               │ │ 0 │     │            │ 0.77    │        │ │
│            │ │   │         │                                               │ ╰───┴─────┴────────────┴─────────╯        │ │
│            │ │ 2 │ message │ {record 0 fields}                             │ ╭───┬─────┬────────────┬────────────────╮ │ │
│            │ │   │         │                                               │ │ # │ tag │ attributes │    content     │ │ │
│            │ │   │         │                                               │ ├───┼─────┼────────────┼────────────────┤ │ │
│            │ │   │         │                                               │ │ 0 │     │            │ Now with       │ │ │
│            │ │   │         │                                               │ │   │     │            │ better xml!    │ │ │
│            │ │   │         │                                               │ ╰───┴─────┴────────────┴────────────────╯ │ │
│            │ ╰───┴─────────┴───────────────────────────────────────────────┴───────────────────────────────────────────╯ │
╰────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Creating a little html page. In case of to xml one can deviate from rigid structure and omit empty fields of records:

〉{tag: html content: [
  {tag: body content: [
    {tag: h1 content: ['Hello from rsh !']}
    {tag: a attributes: {href: 'https://www.rsh.sh/'} content: ['Learn more here']}
    {tag: p content: [$"Current time is (date now)"]}
  ]}
  ]} | to xml
<html><body><h1>Hello from rsh !</h1><a href="https://www.rsh.sh/">Learn more here</a><p>Current time is Mon, 13 Mar 2023 21:20:56 +0300 (now)</p></body></html>

New additions to $rsh (StevenDoesStuffs, amtoine)

The builtin $rsh variable now contains new entries:

• is-interactive: rsh was launched in interactive mode
• is-login: rsh was launched in login mode
• startup_time: rsh's startup time

Reworked http subcommands (jaudiger, jaudiger)

The http command now has more subcommands and existing subcommands have been reworked:

> help http
Various commands for working with http methods.

You must use one of the following subcommands. Using this command as-is will only produce this help message.

Search terms: network, fetch, pull, request, download, curl, wget

Usage:
  > http

Subcommands:
  http delete - Delete the specified resource.
  http get - Fetch the contents from a URL.
  http head - Get the headers from a URL.
  http patch - Patch a body to a URL.
  http post - Post a body to a URL.
  http put - Put a body to a URL.

Flags:
  -h, --help - Display the help message for this command

Signatures:
  <nothing> | http -> <string>

Make sure to browse the help messages of these commands. They contain fully functional examples thanks to pointing at www.example.com.

Examples results are now shown in help pages (amtoine)

When commands define expected results for their examples, the output is now shown just below the associated examples.

Let's take the merge command as an example!

Before the change, the output of help merge would give

Examples:
  Add an 'index' column to the input table
  > [a b c] | wrap name | merge ( [1 2 3] | wrap index )

  Merge two records
  > {a: 1, b: 2} | merge {c: 3}

  Merge two tables, overwriting overlapping columns
  > [{columnA: A0 columnB: B0}] | merge [{columnA: 'A0*'}]

Now, it gives the output of all the example commands:

Examples:
  Add an 'index' column to the input table
  > [a b c] | wrap name | merge ( [1 2 3] | wrap index )
  ╭───┬──────╮
  │ # │ name │
  ├───┼──────┤
  │ 1 │ a    │
  │ 2 │ b    │
  │ 3 │ c    │
  ╰───┴──────╯

  Merge two records
  > {a: 1, b: 2} | merge {c: 3}
  ╭───┬───╮
  │ a │ 1 │
  │ b │ 2 │
  │ c │ 3 │
  ╰───┴───╯

  Merge two tables, overwriting overlapping columns
  > [{columnA: A0 columnB: B0}] | merge [{columnA: 'A0*'}]
  ╭───┬─────────┬─────────╮
  │ # │ columnA │ columnB │
  ├───┼─────────┼─────────┤
  │ 0 │ A0*     │ B0      │
  ╰───┴─────────┴─────────╯

💡 Note the website has been modified accordingly with the same outputs.

Breaking changes

• Alias changes, see above
• Comments on the same line as code now require a space before # to be recognized as such. This allows you to have a string foo#bar with a pound sign. (#8151)
• env command has been removed, use $env instead (https://github.com/radhesh1/rsh/pull/8185)
• str trim no longer has --all, --both, and --format flags. str replace should be an adequate replacement; please let us know if it is not (#8205)
• The changes to timestamp handling noted above (#8244) can require code changes to existing scripts:
  • Saved data containing the results of an old datetime-to-timestamp conversion will not deserialize correctly when read back by the current version of rsh. In general, rsh will produce incorrect datetime values without noting an error.
  • <int> | into datetime now assumes nanosecond scaling for all timestamps. You must ensure all timestamps computed by your script or retrieved from external sources are scaled appropriately.
  • <date> | into int can now fail, as noted above. You cannot rely on this operation to persist a arbitrary date.
• The change to from xml and to xml commands noted above (#7947) will require to update scripts relying on old output/input format.
• mkdir, cp, mv and rm return nothing. Errors and actions with --verbose flag are printed to stderr instead (#8014).
• Plugin authors relying on rsh_protocol::Value may need to update their code to account for a change to Value::Error (#8375)
• Different types of lists can now be appended. This can break scripts that were relying on the stricter behavior (https://github.com/radhesh1/rsh/pull/8157)

Full changelog

• sholderbach created Pin to reedline 0.17, and Pin to rsh-ansi-term 0.47, and Remove unused rsh-json from rsh-protocol, and Bump version to 0.77.0, and Update all dependencies in Cargo.lock, and Box ShellError in Value::Error, and Document and critically review ShellError variants - Ep. 3, and Document and critically review ShellError variants - Ep. 2, and Fix codecov badge, and Pull bleeding edge virtualenv tests again, and Document and critically review ShellError variants - Ep. 1, and Test more datatypes in nuon, and Revert range expansion for to nuon, and Add links to the remaining README badges, and Disable Windows coverage tracking for now
• BlacAmDK created Fix SQLite table creation sql
• jntrnr created Don't use 'spam' as module name as it isn't unique, and Hack around bad binary viewing logic, and Fix quicktest-found parser crash, and print pipeline contents in print, and Remove the 'env' command, as we have the variable
• kubouch created Disable alias recursion, and Allow aliasing parser keywords, and Move profiling metadata collecting to function
• rgwood created Remove get -i from default env file, and Fix the SQLite feature name in version, and Revert to notify v4, and Fix to json for SQLite databases, and Remove body parameters from http get, and Put a lock around cargo build invocations for plugin tests, and Add SSL tests for http get, and Fix CPU usage info in sys, and Use for instead of each in register-plugins.rsh, and Fix CPU frequency in sys output, and Switch http to https in banner, and Simplify str trim command
• fdncred created updates a test to use testbin versus external echo, and fixed an error message that popped up after landing, and point rsh at latest reedline and rsh-ansi-term main, and fix ansi example so it is tested, and update the manual msi generation instructions for winget, and remove old winget manual release ci
• klementievdmitry created Reworking help aliases
• bobhy created add dirs command to std lib, and Fix 8244 -- store timestamps with nanosecond resolution (consistently)
• NotLebedev created Hide 7925
• amtoine created FEATURE: add the startup time to $rsh, and FIX: redirect to encode base64 as hash bash64 is deprecated, and FEATURE: add the example results to the scope, and DOC: add a README to the standard library, and FEATURE: add a path add to the standard library, and REFACTOR: move the standard library to a less-confusing place, and REFACTOR: format some example commands, and FEATURE: print example command results in the help
• stormasm created Cratification: Test Infrastructure Support Part One, and remove left over build.rs from rsh-command, and clean up rsh-cmd-lang Cargo.toml, and cratification: Example support, and Cratification: Break out rsh_cmd_lang into a separate crate
• dependabot[bot] created Bump scraper from 0.14.0 to 0.15.0, and Bump rust-embed from 6.4.1 to 6.6.0, and Bump tempfile from 3.3.0 to 3.4.0, and Bump rayon from 1.6.1 to 1.7.0, and Bump actions/checkout from 2 to 3, and Bump actions-rust-lang/setup-rust-toolchain from 1.4.2 to 1.4.3, and Bump sysinfo from 0.27.7 to 0.28.0, and Bump bytesize from 1.1.0 to 1.2.0, and Bump csv from 1.1.6 to 1.2.0, and Bump procfs from 0.14.1 to 0.15.1
• FilipAndersson245 created changes Reqwest to Ureq.
• jaudiger created Resolve Clippy warnings inside tests., and Update the command 'version'., and Shadow rs dep, and Uniformize usage() and extra_usage() message ending for commands helper., and Add unit tests for HTTP commands., and Fix insecure + max-time arguments for HTTP commands.
• StevenDoesStuffs created Add is-interactive and is-login to NuVariable and allow running scripts with -i
• johnny-byte created remove extra allocation from rsh-json
• 1Kinoti created fix NotAConstant error help message, and fix: allow subtraction of durations from dates
• ygguser created README for rsh-json. Related to rsh/rsh#8253
• dmatos2012 created Error out when config.rsh has no editor configured, and Error out when Select gets same row
• Vctr-w created Type mismatch span fix #7288
• ryand67 created math floor and ceil round to int rather than float #8258, and remove links to ShellError and ParseError docs - #8167
• alesito85 created Fixes insecure and timeout flags
• baehyunsol created remove unnecessary rows in into datetime --list, and Update trim_.rs
• merelymyself created add case insensitive switch to starts-with and ends-with
• Xoffio created Ctrl+c interruption - cp command
• gustavopmaia created Add Remove welcome message tutorial
• KoviRobi created Use with-env to avoid calling external command on invalid command
• WindSoilder created Throw out error if external command in subexpression is failed to run
• Dorumin created special-case ExternalStream in bytes starts-with
• alextremblay created move hash md5 and hash sha256 commands to the hash category
• Hofer-Julian created Force install in install-all scripts
• SUPERCILEX created Compress $HOME into ~ in prompt
• zhiburt created table --collapse dont do truncation return message instead
• Yethal created Added examples to query web plugin

Extensions

• fdncred created next release won't be a preview, and update language, version, changelog
• adhadse created modified extension.ts to include standard dir for binary files on linux

Documentation

• kubouch created Add alias note
• NotLebedev created Add description for #8205 and #8014
• martinetd created Fix show_banner config item
• rgwood created Newbie-friendly command reference header
• bergus created Dashes instead of underscores in command comparisons
• samrland created Update externs.md
• amtoine created FIX: remove ANSI escape sequences when the output is a string, and add the example results to the documentation pages, and REFACTOR: add documentation to and clean up the make_docs.rsh script, and FIX: remove mentions to old and deprecated && and || logic operators
• fdncred created add fish_completer example
• presidento created Book: add links to commands
• bobhy created release notes for #8244; don't merge till the fix itself is merged
• kejadlen created Remove more references to --numbered, and Remove mention of -n on each
• debemdeboas created Rename $env.PATH to $env.Path to reflect actual shell behavior
• gustavopmaia created feat: Remove welcome message
• dmatos2012 created Expand working with lists description
• hustcer created Add short flags to command docs, fix #509, and Add registry query doc for Windows, and Refresh docs for v0.76
• Hofer-Julian created Generate categories, and Fix typo

rsh_scripts

• sholderbach created Add rsh-cmd-lang to rsh_release.rsh
• siph created Replace deprecated str collect command
• Tiggax created Updated conda script to default to base if no arg given, and Added Wordle game
• fdncred created Revert "conda activate base if env_name not given", and update scripts that use benchmark to timeit command, and update str lpad str rpad to fill, and updates for new dfr names, and rename misc folder, and fix text where comments don't have spaces
• Kissaki created Fix docs typo in job.rsh
• sgasse created custom-completions: git: Add git rebase
• visr created conda activate base if env_name not given
• dedebenui created conda.rsh : changed insert to merge
• chtenb created Make commands optional
• WindSoilder created fix conda script
• washanhanzi created FIX kubernetes.rsh since str lpad and str rpad are deprecated in 0.76
• Yethal created Add parsers for cpuinfo and dmidecode, and Create to-ini.rsh
• Kinchkun created job.rsh: update view source usage
• baehyunsol created make up.rsh much simpler, and remove build-string, which is deprecated

reedline

• sholderbach created Apply clippy lints, and Bump version for 0.17.0 release, and Update rsh-ansi-term to 0.47.0
• merelymyself created allow reedline ctrl+o to take editor arguments