learning sed, awk and regex

[u/broken_py]

Looking to learn sed, awk, and regular expressions properly.

What are the best free resources—books, websites, tutorials, courses, or hands-on exercises—you’d recommend?

Also, any good book recommendations for learning these tools from beginner to advanced?

[u/DanceHackRock]

man perlre

That's the documentation for regular expressions you need.

[u/Kitchen-Angle-2401]

for awk and sed and other few essential bash tool ... finish hackerrank's linux shell tasks ... https://www.hackerrank.com/domains/shell

[u/moucheh-]

https://regex101.com https://regexone.com

Go through the exercises on these web sites

[u/michaelpaoli]

Perhaps a bit unconventional, but I'd quite suggest:

• Learn it bit-by-bit, mostly following historical trail, practicing and well and rather solidly learning each piece, before moving on to the next. That will also have key advantages, notably helping one to better learn not only some origins and history, but more notably what does and doesn't apply to BRE, ERE, PRE, and at least some key/common noteworthy exceptions along the way. And to that end, I'd recommend learning in this order - and mostly focused on REs:
• start with the basic shell globbing, just the basic POSIX stuff: *?[] - just those bits, no more. Though most won't even think of that as RE, technically it is also RE, and good simple start, and yes, does have some relevant overlap as one goes from there and then further to BRE, etc.
• BRE - start as concisely(/simple? ;-)) as possible, but no simpler. Yes, ed ... ye olde ed, from UNIX, 7th edition, notably the man page thereof. It exceedingly well covers one of the earliest BRE definitions and full specifications, in very concise form (also reproduced further below, for reference). And while one is at it, well learn ed, as it also shares much in common with sed (and ex/vi). After mastering BRE in ed (one needn't use ancient ed itself, a modern ed will work perfectly fine to learn and practice such), then well read over the specifications for BRE in current POSIX. It adds a wee bit, but not much (mostly a whole lot more words to cover the same stuff, but some modest bits added). For practice, one can not only well use ed, but also grep (but without -E nor -F options, nor fgrep nor egrep), and sed (though sed very slightly extends such). And I'd generally recommend stick with POSIX features and definitions, and avoid use of other (e.g. GNU) extensions, to generally avoid confusion (at least at this point). Can also use vi likewise too (but avoid vim's extensions to BRE - as it makes a whole helluva lot of exceptions - so again, recommended stick to POSIX - e.g. nvi or BSD's vi, if feasible). Well mastered BRE? Should be able to figure out how to, e.g. use ed or grep to find, in file /usr/dict/words (very long list of words, one per line), all words which are five character palindromes (same forwards and backwards) in that file. For even more of a challenge, do a more proper definition of palindromes - having so matched exactly five letters, ignoring all other characters in the line - and to be fully complete, ignore case (might need to use a non-ancient grep to well do that, and one capability it has beyond what BRE itself covers). And get highly good at BRE and sed, one can, e.g. implemented Tic-Tac-Toe in sed, as I did ... sed is, after all, also a Turing Complete programming language.
• ERE - after having mastered BRE, on to ERE - ERE modestly expands BRE. If one wants to do it historically, look at AWK or egrep - yeah, all the way back to 1979 UNIX Seventh Edition - or simply read the POSIX definition of ERE. And then well practice using egrep / grep -E, and (as POSIX as feasible) awk. And, well, if one wants to well learn awk, it not only uses ERE, but is a quite capable (though rather specialized) programming language, that in general goes well beyond most of sed's programming capabilities. And after having mastered ERE (should be pretty easy after mastering BRE, as ERE doesn't add all that much to BRE) ...
• PRE. Highly well study the relevant man page (perlre(1)). As for practice, one can use perl, or other languages that more-or-less incorporate PRE, generally with little to minor modifications, e.g. per, python, Java, PHP, among many others. It will take a while to master PRE - there's a whole lot to it, and very powerful. For the most part, if one can even conjure up a problem for which RE may be the solution, for the most part, there's a way to do it with PRE (okay, not absolutely everything can be done/solved with PRE, though dang close ... and if one then combines that with a programming language, like Perl, well, then can probably cover anything and everything that might need to be done/solved). After highly well learning that, then also learn the distinctions between PRE1 and PRE2 (later versions of perl added a moderate bit more stuff). Also learn other (more-or-less or de facto (common) "standards" of PRE or minor variations thereof. E.g. Google has their own implementations thereof, that are slight variations (e.g. most all of it, but stripped of some bits that tend to be highly inefficient algorithmicly, and thus often quite inefficient (for machines/CPUs) in practice, and potentially problematic in that regard). Anyway, having well learned that/those, and the common distinctions among common flavors of REs, and some common/noteworthy exceptions among implementations of their distinct flavors, and being able to practically solve most if not all RE challenges, by then, one will have well mastered REs.

Oh, also, GNU. Since that's a common implementation of REs, should familiarize oneself with GNU's implementation of BREs and EREs. Note also that, for better and/or worse, GNU does deviate (mostly "extend"), a moderate bit beyond what POSIX specified for BRE and ERE. For portability and to avoid unpleasant surprises, I strongly recommend stick to POSIX for BRE and ERE (and GNU will also handle that just fine), lest one possibly get bitten - possibly very hard and unexpectedly (or someone else gets bitten hard by what you left behind ... and then maybe they look for someone to blame or take it out on ... or their heirs, or ...). Most notably GNU plays very loosey-goosey between BRE and ERE, mostly allowing to use either in either context, just by varying the syntax ... so, yeah, generally don't do that - that's highly GNU, and absolutely not POSIX, and will typically break outside of GNU implementations. And no, even GNU's POSIX options/settings won't protect one (at least not fully) from that non-POSIX behavior of GNU (though in some cases it might technically qualify as POSIX, as being "unspecified" - but one should not rely upon unspecified behaviors!).

Ah, history. :-) And yes, my first introduction to BREs - when they were, back then, simply called "regular expressions" (and EREs, "extended regular expressions"). Folks oft complain how complex, etc. Well, here we have a quite full specification for BRE - in a mere 36 lines, 318 "words", 1,854 characters. But do read it carefully and thoroughly, it's quite densely packed. E.g. the specification doesn't add fluff, one could even call it highly compressed (doesn't use redundancy), and uses self-referential forward references, back references, and recursion (hey, written by programmers!), in parts of the specification, so one should read it very carefully - and even quite repeatedly. But it is a whole lots of useful information and a quite complete specification, squeezed down to very little text, with essentially no excess. Once one highly well understands all of it, one will be well on the way to well undersanding BRE:

Ed supports a limited form of regular expression notation.
A regular expression specifies a set of strings of charac- 
ters.  A member of this set of strings is said to be matched
by the regular expression.  In the following specification
for regular expressions the word `character' means any char-
acter but newline.
1.   Any character except a special character matches
     itself.  Special characters are the regular expression
     delimiter plus \[.  and sometimes ^*$.
2.   A . matches any character.
3.   A \ followed by any character except a digit or ()  
     matches that character.
4.   A nonempty string s bracketed [s] (or [^s]) matches any
     character in (or not in) s. In s, \ has no special  
     meaning, and ] may only appear as the first letter.  A
     substring a-b, with a and b in ascending ASCII order,
     stands for the inclusive range of ASCII characters.
5.   A regular expression of form 1-4 followed by * matches
     a sequence of 0 or more matches of the regular expres-
     sion.
6.   A regular expression, x, of form 1-8, bracketed \(x\)
     matches what x matches.
7.   A \ followed by a digit n matches a copy of the string
     that the bracketed regular expression beginning with  
     the nth \( matched.
8.   A regular expression of form 1-8, x, followed by a reg-
     ular expression of form 1-7, y matches a match for x   
     followed by a match for y, with the x match being as
     long as possible while still permitting a y match.
9.   A regular expression of form 1-8 preceded by ^ (or fol-
     lowed by $), is constrained to matches that begin at
     the left (or end at the right) end of a line.
10.  A regular expression of form 1-9 picks out the longest
     among the leftmost matches in a line.
11.  An empty regular expression stands for a copy of the   
     last regular expression encountered.

That's it, no more, no less. A quite complete BRE specification within just that small bit of text!

As for (free) resources, I'd mostly suggest:

• relevant man pages and POSIX documentation
• https://www.mpaoli.net/~michael/unix/regular_expressions/
• https://regex101.com/
• r/regex
• one can also generally find lots of articles, tutorials, books (borrow from library for free), etc. on REs - but do beware, that quality does vary significantly (especially among random bits on 'da Interwebs). E.g. that challenge/exercise/"trick" of finding 5 character/letter palindromes, I ran across well over a quarter century ago in an article in a tech periodical (put out strictly on ye olde dead tree material, back in the day).

[u/Original-Active-6982 → u/michaelpaoli]

I hate to argue with a very comprehensive list of steps to take - and I applaud your discussion! I have been using the PCRE for 30+ years in a huge variety of tasks and am always amazed by some feature I didn't know about (remember).

That's why I recommend a huge "look-ahead" of features that might be useful. Always priming your knowledge pump with things that could help in some future task. Even for very experienced programmers. Keep on learning stuff you don't use and didn't know about.

[u/Original-Active-6982 → u/Original-Active-6982]

Having meant to add this before: And REFACTOR. Always look at how something was implemented in the early phases of a project and think if it makes sense to rewrite some/all of it. Mainly for maintainability and correctness; sometimes for performance.

[u/michaelpaoli → u/Original-Active-6982]

Sure, could well argue for and/or against such. Such looking ahead and preview, might help one better anticipate such, keep broader framework and possibilities in mind, etc. And/or may just add quite a bit more to the confusion and make the learning more difficult/challenging. Dear knows which approach is best, or what would make the best balance. Also, what's optimal and what techniques work "best", may be quite specific to the individual learner.

In any case, yeah, thanks, you do bring up a good point.

[u/bigkahuna1uk → u/michaelpaoli]

Thanks for a great informative answer
https://giphy.com/gifs/rY93u9tQbybks

[u/EnzymesandEntropy]

This blog post on "idiomatic awk" really helped awk finally click for me https://backreference.org/2010/02/10/idiomatic-awk/

[u/outer-pasta]

I am interested in this question also, for historical purposes.

I wanted to note that learning how to use the ed text editor really does provide some interesting historical context, you can learn it with info ed. The design of regex and sed and grep make much more sense if you know how to use ed, just one example is the names: s+ed is stream editor, g+re+p is global/regex/print.

ed is the standard Unix text editor.

[u/mjmvideos → u/outer-pasta]

Ed, man! !man ed

[u/michaelpaoli → u/mjmvideos]

Hell yeah! At least ed, from UNIX Seventh Edition, circa 1979, is a darn good place to start for BRE!

[u/faultydesign]

regex101.com if you want a good web regex debugger

[u/Astro_indie]

I've been reading a pdf from catalunia, that pdf remember me  the basic offline method, info 《command》 , man 《command》, and help

[u/Europia79]

https://www.grymoire.com/Unix/Regular.html — This guy briefly outlines a quick & dirty history of regular expressions. Like, prior to that, I knew that different tools had slightly different syntax's and features—but until I read his explanation, I never knew WHY.

Here's an alternative tutorial: https://www.regular-expressions.info/tutorial.html.

And here's some tips for Bash specific regular expressions (using "${BASH_REMATCH[@]}"): https://kodekloud.com/blog/regex-shell-script/

[u/SpecialistJacket9757]

I learned regex simply by spending hours at the terminal testing variations of [[ "$str" =~ $regex ]] && ${BASH_REMATCH[0]}

It was a great learning experience and I quickly developed expertise in using bash regex

[u/Different-Depth4116]

grymoire.com/Unix/

[u/Slinkinator]

Edx unix tools delftx, diomidis spinellis. 0-100% shell skills in easy steps. I know this falls on deaf ears, but it's still true :(

https://learning.edx.org/course/course-v1:DelftX+UnixTx+1T2022/home

[u/Crushcode8]

add xarg in there :)

[u/Ok-Seaworthiness-542]

I honestly just happy to see regex being used in more products

[u/Radamand]

I remember learning (and using) Awk, I love that tool/language!
It's pretty incredible the things you can do with it.

[u/ASIC_SP]

I have written plenty on these topics:

Ebooks with lots of examples and exercises: https://learnbyexample.github.io/learn_gnused/ and https://learnbyexample.github.io/learn_gnuawk/

BRE/ERE regex cheatsheet and subtle differences between tools: https://learnbyexample.github.io/gnu-bre-ere-cheatsheet/

Interactive TUI apps for exercises: https://github.com/learnbyexample/TUI-apps/blob/main/SedExercises and https://github.com/learnbyexample/TUI-apps/blob/main/AwkExercises

Happy learning :)

[u/Bob_Spud]

I've the first edition which an excellent resource for AWK, the second edition should be an improvement. The AWK Programming Language, 2nd Edition.

[u/dvamnedbouchsleep]

regex is like magic but with more typing

[u/Grouchy-Car-3711]

Refer to the docs pages of those technologies those are the best resources you can find

[u/ftonneau]

Among classic books, Dougherty & Robbins' sed & awk is excellent--the best book on these topics, including regular expressions.

For books more specifically about Awk, I do not recommend Robbins' Effective awk programming as a first reading--the book is chock full of information on different Awk variants, but I found its organization somewhat confusing.

Go instead for Aho, Kernighan & Weinberger's The AWK programming language--amazing book in classic Kernighan style, not only about Awk, but also about programming topics in general (including parsing and algorithms). I cannot recommend this book highly enough.

Then, and only then, pick Robbins' Effective awk programming if you want to refine your knowledge of Awk.

[u/Round_End_6145]

I learned from this book several years ago - https://www.amazon.com/dp/013937681X?lv=shuf&channelId=500&plpRedirect=mhFallback

[u/Rinku_Kurora]

regex101.com

[u/readparse]

Those tools are all useful. But since Perl was created to be all three, and became much more, might as well take a look at that.

[u/Astro_indie → u/readparse]

If perl is not installed?  What we do without all of this? 

[u/StatementOwn4896 → u/Astro_indie]

I’d be surprised if you don’t have Perl installed already

[u/readparse → u/Astro_indie]

Well that's one reason why Perl is relevant to what you're trying to do. It's ubiquitous. It predates pretty much everything else and was extremely popular when Linux started to become popular. So it's generally a safe bet that any Unix-like system has it installed already.

A couple of exceptions: RHEL stopped installing it by default, I think. It's still available, but you asked about already being installed.

And I doubt you're on Cygwin, but they also stopped installing it by default.

Given your question, I'm going to go out on a limb and ponder aloud whether you are using a system that you don't administer, and so you're stuff with what's already on it. I still think the odds are that it has perl on it, but you might have found yourself on a RHEL system that doesn't have it.

I still think what you're looking for may very well be best answered by Perl, but I wouldn't disuade anyone from knowing both sed and awk, because they are both useful in their own ways.

You also mentioned regex, which is less about learning a single command. It's an important topic in all of programming, and one thing you may run across is something called PCRE, which stands for Perl Compatible Regular Expressions which is still (I think) the gold standard of regex support.

Why? Because Perl took regular expressions to a whole new level. So much so that somebody created an open source library specifically for supporting regular expressions in the same way that Perl does, so that people who use languages other than Perl can have that same support.

[u/Astro_indie → u/readparse]

I forgot perl for many years, idk the similitud with gnu/linux, i neef more bit foood xD

[u/Select-Sale2279 → u/Astro_indie]

perl is to linux what water is to the ocean. both exist if one of them do. lol

[u/Not_A_Red_Stapler → u/readparse]

I actually agree with this. You should definitely take a look at Perl OP.

[u/Marques1236]

Se falar português, procure o canal do Professor Blau Araújo no YT.

[u/-lousyd]

If you're looking for a book, O'Reilly's Classic Shell Scripting really cannot be beat for comprehensiveness and clarity. I came to it already having pretty decent skills and it helped me kind of round them out and hone them. It covers everything and it's so so helpful.