9.7. Examples: Reading specific columns from a file or a list of files

Print the first column of each file In order to print the first column of these files, we will use awk utility. As it was shown before, the command for this utility consists on some optional flags followed by an action statement and then the list of files. In this case the action statement is '{print $1}', because we want to print only the first column ($1). Files file2.txt and file4.txt use a space as a column separator (which is the separator for default). So, to access the first column of these files we don't need the -F flag. However, files file1.csv and file3.csv use a comma as a separator. So, in order for awk to distinguish the different columns we have to use the -F flag. In this case, the parameter of the -F flag is a comma (-F','). If you are trying to read a file which has the columns separated by a different character, then use that character instead of ','.

To precede each line by the line number, add NR, after the string print in the awk command to indicate that you want to print the Number Row before the column 1 ($1):

Print the first column of file1.csv and file2.txt in reverse order In order to print from the last line to the first line, you can use the command tail with the flag -r (for reverse) after the command awk. The command line will first execute the awk command which is written before the | symbol, and then it will run the tail command which inverts the order of the previous output. Remember that for file1.csv you need to use -F',' to indicate that the columns are separated by commas and not spaces.

The same as in example 1, to precede each line by the line number, add NR, after the string print in the awk command to indicate that you want to print the Number Row before the column 1 ($1):

Print the second and third columns of file1.csv and file2.txt. Show the lines of file2.txt in reverse order. In the previous examples we used the action statement '{print $1}' to print the first column. Since we now want to print the second and third columns instead of the first one, we replace $1 by $2, $3. If you wanted to print column 4 and 5 instead, then you would simply use $4,$5, etc. In order to print the output in reverse order for file2.txt, use the tail -r command after the awk.

To precede each line by the line number, add NR, after the string print in the awk command to indicate that you want to print the Number Row before the column 1 ($1):

Print all the columns of file1.csv showing the lines in reverse order. To print all the columns of a file using awk simply use $0 (instead of a column number). Or use the command cat.

Print all the columns of file1.csv in reversed order (first the third column, then the second and finally the first one), and save the re-ordered columns in a new file called file1_reordered.csv. If you were going to print the columns one to three in normal order, you would use '{print $1,$2,$3}'. To print them in reverse order, you just reverse the order of the columns in print: '{print $3,$2,$1}'.
To save the output to a file instead of showing it in the terminal, use >> file as explained in previous sections.
Remember to use the -F',' flag to indicate that the columns are separated by commas and not the default space.

$ awk -F',' '{print $3,$2,$1}' file1.csv >> file1_reordered.csv
$ cat file1_reordered.csv

"HASCONDITION" "Subject Group" "Anonymized ID"
"0" "Group1" "B33199522"
"0" "Group3" "B33199603"
"0" "Group1" "B11137879"
"0" "Group2 b" "B11144410"
"0" "Group2 b" "B11110455"
"0" "Group3" "B11135291"
"0" "Group1" "B11153927"
"0" "Group2 b" "B11177579"
"MD" "Group1" "B11177806"
"0" "Group3" "B11157958"
"0" "Group3" "B11110690"
"0" "Group1" "B11152799"
"0" "Group1" "B11154358"
"0" "Group1" "B11110925"
"9" "Group3" "B11135291"
"0" "MISSING" "B11135072"
"0" "Group3" "B33199603"
"0" "Group1" "B11137879"
"0" "Group1" "B11110603"
"0" "Group1" "B11110927"
"0" "Group1" "B11147712"
"0" "Group2 b" "B33191224"
"0" "Group2 b" "B11131290"
"0" "Group1" "B11157974"
"0" "Group2 b" "B33191224"
"0" "Group3" "B11141503"
"9" "Group3" "C11137159"
"0" "Group1" "B33199522"

Print all the columns of file1.csv in reversed order (first the third column, then the second and finally the first one). Print the lines also in reverse order (from the last line to the first one).
Use the same command as before, adding | tail -r at the end to invert also the lines.

$ awk -F',' '{print $3,$2,$1}' file1.csv | tail -r
"0" "Group1" "B33199522"
"9" "Group3" "C11137159"
"0" "Group3" "B11141503"
"0" "Group2 b" "B33191224"
"0" "Group1" "B11157974"
"0" "Group2 b" "B11131290"
"0" "Group2 b" "B33191224"
"0" "Group1" "B11147712"
"0" "Group1" "B11110927"
"0" "Group1" "B11110603"
"0" "Group1" "B11137879"
"0" "Group3" "B33199603"
"0" "MISSING" "B11135072"
"9" "Group3" "B11135291"
"0" "Group1" "B11110925"
"0" "Group1" "B11154358"
"0" "Group1" "B11152799"
"0" "Group3" "B11110690"
"0" "Group3" "B11157958"
"MD" "Group1" "B11177806"
"0" "Group2 b" "B11177579"
"0" "Group1" "B11153927"
"0" "Group3" "B11135291"
"0" "Group2 b" "B11110455"
"0" "Group2 b" "B11144410"
"0" "Group1" "B11137879"
"0" "Group3" "B33199603"
"0" "Group1" "B33199522"
"HASCONDITION" "Subject Group" "Anonymized ID"

Read the second column of file1.csv and file2.txt and save it into an array. When saving a column of a file into an array, you must specify that the elements of the array are separated by new lines ('\n'). You do this using the command IFS=$'\n'.
The elements of the array will be saved in the variable ARRAY. As it was learned in previous chapters, to access the individual elements of ARRAY you use the syntax ${ARRAY[index]}. With index starting at 0. So, to access the first element the command is echo ${ARRAY[0]}, to access the second element echo ${ARRAY[1]}, etc. Type echo ${ARRAY[@]} to view all elements and to obtain the number of elements (size of ARRAY).
Remember, the system variable IFS contains the separator that is being used to separate each feld within the lines of a file. You can change the value of this variable at any time: IFS='character'

Print the first column of file2.txt followed by the first column of file4.txt. To print a specific column for more than one file, you use the same command, adding the list of files you want to print after the first one. However, all the files in the list must use the same column separator (in this case is a space). Since the column separator for this list of files is a space (the default), you don't need to use the -F flag.

$ awk '{print $1}' file2.txt file4.txt
"AnonymizedID"
"B11130912"
"B11137244"
"B11154534"
"B11144100"
"B11137244"
"B12226566"
"B11134987"
"B11144345"
"C11137159"
"B11156453"
"B11110676"
"C11138929"
"B11154532"
"B11155267"
"B11137120"
"B33191224"
"B11155267"
"C11138999"
"B11131605"
"B11137784"
"B11156098"
"B11133232"
"B11135292"
"C11138912"
"B11150911"
"B11152577"
"B11156098"
"B11133232"
AnonymizedID
B11108326
B11110893
B11119909
D11144030
D11144030
B11119903
C11131039
C11133100
C11135566
C11137159
C11137159
C11137167
C11137167
C11137439
C11137439
C11137443
C11137544
C11137123
C11138150
C11138152
C11138797
C11138184
C11138122
C11138122
C11138192
B12226507
B12226546

Print the first column of file3.csv followed by the first column of file1.csv. Since the column separator for this list of files is a comma, you need to use the -F',' flag.

$ awk -F',' '{print $1}' file3.csv file1.csv
Anonymized ID
C11138122
C11138192
B12226507
B12226546
C11138122
C11138184
C11138797
C11138152
C11138150
C11137167
C11137159
C11137167
C11137159
C11131039
C11135566
B11119903
C11137544
C11137443
C11137123
C11137439
C11137439
C11133100
D11144030
B11108399
B11108326
B11119909
B11110893
"Anonymized ID"
"B33199522"
"B33199603"
"B11137879"
"B11144410"
"B11110455"
"B11135291"
"B11153927"
"B11177579"
"B11177806"
"B11157958"
"B11110690"
"B11152799"
"B11154358"
"B11110925"
"B11135291"
"B11135072"
"B33199603"
"B11137879"
"B11110603"
"B11110927"
"B11147712"
"B33191224"
"B11131290"
"B11157974"
"B33191224"
"B11141503"
"C11137159"
"B33199522"