Hacker News Re-Imagined

Bash patterns I use weekly

  • 278 points
  • 7 days ago

  • @gcmeplz
  • Created a post
  • • 111 comments

Bash patterns I use weekly


@penguin_booze 6 days

Replying to @gcmeplz 🎙

If I find myself running a set of commands in parallel, I'd keep a cheap Makefile around: individual commands I want to run in parallel will be written as phony targets:

  all: cmd1 cmd2
  
  cmd1:
      sleep 5
  
  cmd2:
      sleep 10

And then

  make -j[n]

Reply


@stewartbutler 7 days

Replying to @gcmeplz 🎙

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:

    grep -l -r pattern /path/to/files | while read x; do echo $x; done
or the like.

This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arrays or the like.

One place I do use bash for loops is when iterating over args, e.g. if you create a bash function:

    function my_func() {
        for arg; do
            echo $arg
        done
    }
This'll take a list of arguments and echo each on a separate line. Useful if you need a function that does some operation against a list of files, for example.

Also, bash expansions (https://www.gnu.org/software/bash/manual/html_node/Shell-Par...) can save you a ton of time for various common operations on variables.

Reply


@1vuio0pswjnm7 6 days

Replying to @gcmeplz 🎙

To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining").

The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all.

https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-...

Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/1.1, but even a simple shell script will work. Even more, we get easy control over SNI which curl binary does have have.

There are different and more concise ways, but below is an example, using the IFS technique.

This also shows the use of sed's "P" and "D" commands (credit: Eric Pement's sed one-liners).

Assumes valid, non-malicious URLs, all with same host.

Usage: 1.sh < URLs.txt

       #!/bin/sh
       (IFS=/;while read w x y z;do
       case $w in http:|https:);;*)exit;esac;
       case $x in "");;*)exit;esac;
       echo $y > .host
       printf '%s\r\n' "GET /$z HTTP/1.1";
       printf '%s\r\n' "Host: $y";
       # add more headers here if desired;
       printf 'Connection: keep-alive\r\n\r\n';done|sed 'N;$!P;$!D;$d';
       printf 'Connection: close\r\n\r\n';
       ) >.http
       read x < .host;
       # SNI;
       #openssl s_client -connect $x:443 -ign_eof -servername $x < .http;
       # no SNI;
       openssl s_client -connect $x:443 -ign_eof -noservername < .http;
       exec rm .host .http;

Reply


@marcodiego 7 days

Replying to @gcmeplz 🎙

We need a simpler regex format. One that allows easy searching and replacing in source code. Of course, some IDE's already to that pretty well, but I'd like to be able to do it from the command line with a stand alone tool I can easily use in scripts.

The simplest thing I know that is able to do that is coccinelle, but even coccinelle is not handy enough.

Reply


@barbazoo 7 days

Replying to @gcmeplz 🎙

> git bisect is the "real" way to do this, but it's not something I've ever needed

uh, yeah, you did need it, that's why you came up with "2. Track down a commit when a command started failing". Seriously though, git bisect is really useful to track down that bug in O(log n) rather than O(n).

Reply


@harvie 6 days

Replying to @gcmeplz 🎙

curl localhost:8080/{foo,bar,baz}

Reply


@guruparan18 7 days

Replying to @gcmeplz 🎙

I am confused how this works. I would assume `SECONDS` would just be a shell variable and it was first assigned `0` and then it should stay same, why did it keep counting the seconds?

    > SECONDS
    bash: SECONDS: command not found
    > SECONDS=0; sleep 5; echo $SECONDS;
    5
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 41 seconds
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 51 seconds
    > echo "Your command completed after $SECONDS seconds";
    Your command completed after 53 seconds

Reply


@ghostly_s 7 days

Replying to @gcmeplz 🎙

Huh, never knew about $SECONDS.

Reply


@unixhero 7 days

Replying to @gcmeplz 🎙

A lot of genious moves here I have never seen or thought of. Brilliant. Grabbing pids was mindblowingly effective.

Reply


@R0flcopt3r 7 days

Replying to @gcmeplz 🎙

You can use `wait` to wait for jobs to finish.

    some_command &
    some_other_command &
    wait

Reply


@Arch-TK 7 days

Replying to @gcmeplz 🎙

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate.

To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-bisect.

Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides?

If your complaint is that for a in b c d; do ...; done is unclear, then maybe also use lists there, because what's definitely LESS clear is putting things in a variable and relying on splitting.

And most importantly, DO quote things (except when it's unnecessary).

Reply


@michaelhoffman 7 days

Replying to @gcmeplz 🎙

I have something like this in my bashrc:

   preexec ()
   {
       # shellcheck disable=2034
       _CMD_START="$(date +%s)"
   }

   trap 'preexec; trap - DEBUG' DEBUG

   PROMPT_COMMAND="_CMD_STOP=\$(date +%s)
       let _CMD_ELAPSED=_CMD_STOP-_CMD_START

       if [ \$_CMD_ELAPSED -gt 5 ]; then
           _TIME_STR=\" (\${_CMD_ELAPSED}s)\"
       else
           _TIME_STR=''
       fi; "

    PS1="\n\u@\h \w\$_TIME_STR\n\\$ "

    PROMPT_COMMAND+="trap 'preexec; trap - DEBUG' DEBUG"
Whenever a command takes more than 5 s it tells me exactly how long at the next prompt.

I didn't know about `$SECONDS` so I'm going to change it to use that.

Reply


@js2 6 days

Replying to @gcmeplz 🎙

My git-gsr (global search replace) command:

https://gist.github.com/jaysoffian/0eda35a6a41f500ba5c458f02...

Uses perl instead of gsed, defaults to fixed strings but supports perl regexes, properly handles filenames with whitespace.

Reply


@caymanjim 7 days

Replying to @gcmeplz 🎙

Installing GNU stuff with the 'g' prefix (gsed instead of sed) means having to remember to include the 'g' when you're on a Mac and leave it off when you're on Linux, or use aliases, or some other confusing and inconvenient thing, and then if you're writing a script meant for multi-platform use, it still won't work. I find it's a much better idea to install the entire GNU suite without the 'g' prefix and use PATH to control which is used. I use MacPorts to do this (/opt/local/libexec/gnubin), and even Homebrew finally supports this, although it does it in a stupid way that requires adding a PATH element for each individual GNU utility (e.g. /usr/local/opt/gnu-sed/libexec/gnubin).

Reply


@jph 7 days

Replying to @gcmeplz 🎙

> git bisect is the "real" way to do this, but it's not something I've ever needed

git bisect is great and worth trying; it does what you're doing in your bash loop, plus faster and with more capabilities such as logging, visualizing, skipping, etc.

The syntax is: $ git bisect run <command> [arguments]

https://git-scm.com/docs/git-bisect

Reply


@bloopernova 7 days

Replying to @gcmeplz 🎙

This thread seems like a good place to ask this:

When you're running a script, what is the expected behaviour if you just run it with no arguments? I think it shouldn't make any changes to your system, and it should print out a help message with common options. Is there anything else you expect a script to do?

Do you prefer a script that has a set of default assumptions about how it's going to work? If you need to modify that, you pass in parameters.

Do you expect that a script will lay out the changes it's about to make, then ask for confirmation? Or should it just get out of your way and do what it was written to do?

I'm asking all these fairly basic questions because I'm trying to put together a list of things everyone expects from a script. Not exactly patterns per se, more conventions or standard behaviours.

Reply


@geocrasher 7 days

Replying to @gcmeplz 🎙

I want to like this, but the for loop is unnecessarily messy, and not correct.

   for route in foo bar baz do
   curl localhost:8080/$route
   done
That's just begging go wonky. Should be

  stuff="foo bar baz"
  for route in $stuff; do
  echo curl localhost:8080/$route
  done
Some might say that it's not absolutely necessary to abstract the array into a variable and that's true, but it sure does make edits a lot easier. And, the original is missing a semicolon after the 'do'.

I think it's one reason I dislike lists like this- a newb might look at these and stuff them into their toolkit without really knowing why they don't work. It slows down learning. Plus, faulty tooling can be unnecessarily destructive.

Reply


@Hikikomori 7 days

Replying to @gcmeplz 🎙

My daily bash pattern.

  cat file
  cat file | grep something

Reply


@gnubison 7 days

Replying to @gcmeplz 🎙

It’s simple enough to use macos sed here instead of needing gsed: just use ‘-i ''’ instead of ‘-i’.

Reply


@usefulcat 7 days

Replying to @gcmeplz 🎙

> Use for to iterate over simple lists

I definitely use this all the time. Also, generating the list of things over which to iterate using the output of a command:

    for thing in $(cat file_with_one_thing_per_line) ; do ...

Reply


@notatoad 6 days

Replying to @gcmeplz 🎙

I really love this style of blog post. Short, practical, no backstory, and not trying to claim one correct way to do anything. Just an unopinionated share that was useful to the author.

It seems like a throwback to a previous time, but honestly can't remember when that was. Maybe back to a time when I hoped this was what blogging could be.

Reply


@l0b0 7 days

Replying to @gcmeplz 🎙

  > 1. Find and replace a pattern in a codebase with capture groups
  > git grep -l pattern | xargs gsed -ri 's|pat(tern)|\1s are birds|g'
Or, in IDEA, Ctrl-Shift-r, put "pat(tern)" in the first box and "$1s are birds" in the second box, Alt-a, boom. Infinitely easier to remember, and no chance of having to deal with any double escaping.

Reply


About Us

site design / logo © 2021 Box Piper