I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:
grep -l -r pattern /path/to/files | while read x; do echo $x; done
or the like.
This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arrays or the like.
One place I do use bash for loops is when iterating over args, e.g. if you create a bash function:
function my_func() {
for arg; do
echo $arg
done
}
This'll take a list of arguments and echo each on a separate line. Useful if you need a function that does some operation against a list of files, for example.
Once I discovered functions are pipeline-friendly, I started pipelining all the things. Almost any `for arg` can be re-designed as a `while read arg` with a function.
Here's what it can look like. Some functions I wrote to bulk-process git repos. Notice they accept arguments and stdin:
# ID all git repos anywhere under this directory
ls_git_projects ~/src/bitbucket |
# filter ones not updated for pre-set num days
take_stale |
# proc repo applies the given op. (a function in this case) to each repo
proc_repos git_fetch
> I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:
for loops were exactly the pain point that lead me to write my own shell > 6 years ago.
I can now iterate through structured data (be it JSON, YAML, CSV, `ps` output, log file entries, or whatever) and each item is pulled intelligently rather than having to conciously consider a tonne of dumb edge cases like "what if my file names have spaces in them"
eg
» open https://api.github.com/repos/lmorg/murex/issues -> foreach issue { out "$issue[number]: $issue[title]" }
380: Fail if variable is missing
379: Backslashes and code comments
378: Improve testing facility documentation
377: v2.4 release
361: Deprecate `swivel-table` and `swivel-datatype`
360: `sort` converts everything to a string
340: `append` and `prepend` should `ReadArrayWithType`
Powershell is also a good option nowadays (although a lot of people on HN seem to dismiss it for various, imo rather superficial, reasons).
PS> (irm https://api.github.com/repos/lmorg/murex/issues) | % { echo "$($_.number): $($_.title)" }
380: Fail if variable is missing
379: Backslashes and code comments
378: Improve testing facility documentation
377: v2.4 release
361: Deprecate `swivel-table` and `swivel-datatype`
360: `sort` converts everything to a string
340: `append` and `prepend` should `ReadArrayWithType`
Or just
PS> (irm https://api.github.com/repos/lmorg/murex/issues) | format-table number, title
number title
------ -----
380 Fail if variable is missing
379 Backslashes and code comments
378 Improve testing facility documentation
377 v2.4 release
361 Deprecate `swivel-table` and `swivel-datatype`
360 `sort` converts everything to a string
340 `append` and `prepend` should `ReadArrayWithType`
Or even `(irm https://api.github.com/repos/lmorg/murex/issues) | select number, title | out-gridview`, which would open a GUI list (with sorting and filtering), but I think that only works on Windows.
The reason I dismissed Powershell was that it doesn't always play nicely with existing POSIX tools, which is very much not a superficial reason :)
Murex aims to give Powershell-style types but still working seamlessly with existing CLI tools. An attempt at the best of both worlds. But I'll let others be the judge of that.
It's also worth noting that Powershell wasn't available for Linux when I first built murex so it wasn't an option even if I wanted it to be.
It’s a trade off. I had to use a piped loop earlier this year to extract fallout new Vegas mods and textures since they all had spaces in their file names. For this it was perfect to pipe a list of the names to a loop, but for 99% of things I just use a for loop
BTW, the problem I mentioned earlier can be avoided by using `< <()`:
$ x=1
$ seq 5 | while read n; do (( x++ )); done
$ echo $x
1
$ while read n; do (( x++ )); done < <(seq 5)
$ echo $x
6
Almost makes me wonder what the benefit of preferring a pipe here is. I guess it's just about not having to specify what part of the pipeline is in the same shell.
It’s funny I’ve been using Linux for a decade and a half, professionally for about half that time, and yet I still go to python when arithmetic is involved. I’ve been learning a lot about the shell lately it’s like I did the bare minimum with bash just to be able to run programs and slightly automate things and it took this long for it to click with me that it’s a productive programming language in its own right (and probably faster than python.)
BTW you can make it work in bash by setting shopt -s lastpipe. It runs the last part of the pipeline in the main shell, so the mutation variables of will persist.
For me I always used for loops and only recently (after a decade of using Linux daily) have learned about the power of piped-loops. It’s strange to me you are more comfortable with those than for loops, but I think it does make sense as you’re letting a program generate the list to iterate over. A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them using a for loop (this is why I learned about piped loops recently.)
My problem was that I had a directory with probably 200+ subdirectories each one, and the files and subdirectories below them, had a couple of spaces in the name. I typically use
for f in ‘ls’;
For operations like that but it was obviously built on windows (but I run steam on Ubuntu) and I never interact with windows so tbh I had never thought of this problem before.
This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arrays or the like.
One place I do use bash for loops is when iterating over args, e.g. if you create a bash function:
This'll take a list of arguments and echo each on a separate line. Useful if you need a function that does some operation against a list of files, for example.Also, bash expansions (https://www.gnu.org/software/bash/manual/html_node/Shell-Par...) can save you a ton of time for various common operations on variables.